CoChat
Sign Up
PreprintCited by 19Teaching and Learning ProgrammingArtificial Intelligence in EducationOnline Learning and Analytics

The DOI matches its Crossref record.

The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement but Increased Adopters Exam Performances

Open access
Nie, Allen, Chandak, Yash, Suzara, Miroslav, Malik, Ali, Woodrow, Juliette, Peng, Matt, Sahami, Mehran, Brunskill, Emma, Piech, Chris·2024
DOI 10.31219/osf.io/qy8zd
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) are quickly being adopted in a wide range of learning experiences, especially via ubiquitous and broadly accessible chat interfaces like ChatGPT and Copilot. This type of interface is readily available to students and teachers around the world, yet relatively little research has been done to assess the impact of such generic tools on student learning. Coding education is an interesting test case, both because LLMs have strong performance on coding tasks, and because LLM-powered support tools are rapidly becoming part of the workflow of professional software engineers. To help understand the impact of generic LLM use on coding education, we conducted a large-scale randomized control trial with 5,831 students from 146 countries in an online coding class in which we provided some students with access to a chat interface with GPT-4. We estimate positive benefits on exam performance for adopters, the students who used the tool, but over all students, the advertisement of GPT-4 led to a significant average decrease in exam participation. We observe similar decreases in other forms of course engagement. However, this decrease is modulated by the student's country of origin. Offering access to LLMs to students from low human development index countries increased their exam participation rate on average. Our results suggest there may be promising benefits to using LLMs in an introductory coding class, but also potential harms for engagement, which makes their longer term impact on student success unclear. Our work highlights the need for additional investigations to help understand the potential impact of future adoption and integration of LLMs into classrooms.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.31219/osf.io/qy8zd

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Learning improved in programming courses in which students had access to LLMs.

    “Similarly, learning improved in programming courses in which students had access to LLMs (Liu et al. 2024b, Nie et al. 2024).”

    From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Nie et al. 2024

    The Introduction (p. 3) reports an estimated improvement in exam scores for students who adopted and used GPT-4 in one online coding class. The citing sentence broadens that finding to students who merely had access to LLMs in programming courses, without preserving the adopter-only and estimated-outcome qualifications. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Nie et al. (2024) find that the small group of students who do use the LLM performs better on the final exam.

    “Still, the small group of students who do use the LLM performs better on the final exam.”

    From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Nie et al. 2024

    The Introduction (pp. 2–3) reports that only 14.2% of students offered GPT-4 used it and estimates higher exam scores for those adopters. But the exam was an optional diagnostic exam, not a final exam. The 'Exam Score Increased For Adopters' section (p. 11) also describes the benefit as an estimate that was not statistically significant after adjustment for multiple testing. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Nie et al. (2024) observe very limited usage of the LLM and document that it reduces engagement with course materials and likelihood of completing the course.

    “They observe very limited usage of the LLM and document that it reduces engagement with the course materials and the likelihood of completing the course.”

    From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Nie et al. 2024

    The Introduction (p. 2) reports that only 14.2% of students offered the course GPT interface used it. The Results (p. 6) report lower exam participation and week-six homework completion after students were offered and told about GPT-4, supporting reduced course engagement. They do not establish a lower likelihood of completing the course, and the measured effect is of offering and advertising access, not necessarily of using the LLM. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Nie et al. and Kestin et al. implement safeguards against students asking an LLM for solutions to practice problems.

    “Nie et al. and Kestin et al. also implement safeguards against students asking an LLM for solutions to practice problems.”

    From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Nie et al. 2024

  • Previous studies restricted LLMs to certain use cases, in particular by limiting substitutive use.

    “Previous studies restricted LLMs to certain use cases, in particular by limiting substitutive use (Nie et al. 2024, Kestin et al. 2024).”

    From AI Meets the Classroom: When Do Large Language Models Harm Learning? (2024) · cites this paper as Nie et al. 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 33

  1. Angrist, J. D. (1990). Lifetime earnings and the vietnam era draft lottery: evidence from social security administrative records. The american economic review , pages 313-336.
  2. Angrist, J. D., Imbens, G. W., and Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American statistical Association , 91(434):444-455.
  3. Athey, S. and Imbens, G. (2016). Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences , 113(27):7353-7360.
  4. Athey, S. and Imbens, G. W. (2017). The econometrics of randomized experiments. In Handbook of economic field experiments , volume 1, pages 73-140. Elsevier.
  5. Bansak, K. (2020). A Generalized Approach to Power Analysis for Local Average Treatment Effects. Statistical Science , 35(2):254 - 271.
  6. Bigman, M., Roy, E., Garcia, J., Suzara, M., Wang, K., and Piech, C. (2021). Pearprogram: A more fruitful approach to pair programming. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education , pages 900-906.
  7. Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational researcher , 13(6):4-16.
  8. Butgereit, L. and Martinus, H. (2023). Prof pi: Using whatsapp bots and gpt-4 for tutoring mathematics in underserved areas. In International Conference on Innovations and Interdisciplinary Solutions for Underserved Areas , pages 278-289. Springer.
  9. Choi, J. H., Garrod, O., Atherton, P., Joyce-Gibbons, A., Mason-Sesay, M., and Björkegren, D. (2023). Are llms useful in the poorest schools? theteacherai in sierra leone. arXiv preprint arXiv:2310.02982 . · arXiv
  10. Cohen, G. L. and Garcia, J. (2008). Identity, belonging, and achievement: A model, interventions, implications. Current directions in psychological science , 17(6):365-369.
  11. Gemini, Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. (2023). Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 . · arXiv
  12. Gillespie, A. and Hald, J. (2017). The paradox of helping: Contradictory effects of scaffolding people with aphasia to communicate. PLoS One , 12(8):e0180708.
  13. Greene, B. A. and DeBacker, T. K. (2004). Gender and orientations toward the future: Links to motivation. Educational Psychology Review , 16:91-120.
  14. Hanselman, P., Bruch, S. K., Gamoran, A., and Borman, G. D. (2014). Threat in context: School moderation of the impact of social identity threat on racial/ethnic achievement gaps. Sociology of Education , 87(2):106124.
  15. Hansen, J. D. and Reich, J. (2015). Democratizing education? examining access and usage patterns in massive open online courses. Science , 350(6265):1245-1248.
  16. Imbens, G. W. and Angrist, J. D. (1994). Identification and estimation of local average treatment effects. Econometrica , 62(2):467-475.
  17. Kaplan, A. and Flum, H. (2012). Identity formation in educational settings: A critical focus for education in the 21st century. Contemporary Educational Psychology , 37(3):171-175.
  18. Kizilcec, R. F., Piech, C., and Schneider, E. (2013). Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. In Proceedings of the third international conference on learning analytics and knowledge , pages 170-179.
  19. Kizilcec, R. F., Saltarelli, A. J., Reich, J., and Cohen, G. L. (2017). Closing global achievement gaps in moocs. Science , 355(6322):251-252.
  20. Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational researcher , 49(4):241253.
  21. Kumar, H., Rothschild, D. M., Goldstein, D. G., and Hofman, J. M. (2023). Math education with large language models: Peril or promise? Available at SSRN 4641653 .
  22. Leite, W. L., Kuang, H., Shen, Z., Chakraborty, N., Michailidis, G., D'Mello, S., and Xing, W. (2022). Heterogeneity of treatment effects of a video recommendation system for algebra. In Proceedings of the Ninth ACM Conference on Learning@ Scale , pages 12-23.
  23. Liu, R., Zenke, C., Liu, C., Holmes, A., Thornton, P., and Malan, D. J. (2024). Teaching cs50 with ai: leveraging generative artificial intelligence in computer science education. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 , pages 750-756.
  24. Makel, M. C. and Plucker, J. A. (2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher , 43(6):304-316.
  25. Nie, A., Reuel, A.-K., and Brunskill, E. (2023). Understanding the impact of reinforcement learning personalization on subgroups of students in math tutoring. In International Conference on Artificial Intelligence in Education , pages 688-694. Springer.
  26. Noy, S. and Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Available at SSRN 4375283 .
  27. Pardos, Z. A. and Bhandari, S. (2023). Learning gain differences between chatgpt and human tutor generated algebra hints. arXiv preprint arXiv:2302.06871 . · arXiv
  28. Peteranetz, M. S., Flanigan, A. E., Shell, D. F., and Soh, L.-K. (2016). Perceived instrumentality and career aspirations in cs1 courses: Change and relationships with achievement. In Proceedings of the 2016 ACM conference on international computing education research , pages 13-21.
  29. Prihar, E., Lee, M., Hopman, M., Kalai, A. T., Vempala, S., Wang, A., Wickline, G., Murray, A., and Heffernan, N. (2023). Comparing different approaches to generating mathematics explanations using large language models. In International Conference on Artificial Intelligence in Education , pages 290-295. Springer.
  30. Rogers, E. M., Singhal, A., and Quinlan, M. M. (2014). Diffusion of innovations. In An integrated approach to communication theory and research , pages 432-448. Routledge.
  31. Ruan, S., Nie, A., Steenbergen, W., He, J., Zhang, J., Guo, M., Liu, Y., Dang Nguyen, K., Wang, C. Y., Ying, R., et al. (2024). Reinforcement learning tutor better supported lower performers in a math task. Machine Learning , pages 1-26.
  32. Walton, G. M. and Cohen, G. L. (2007). A question of belonging: race, social fit, and achievement. Journal of personality and social psychology , 92(1):82.
  33. Woodrow, J., Malik, A., and Piech, C. (2024). Ai teaches the art of elegant coding: Timely, fair, and helpful style feedback in a global course. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 , pages 1442-1448.

Cited by

Sorted by citations

Metadata

Authors
Nie, A., Chandak, Y., Suzara, M., Malik, A., Woodrow, J., Peng, M., Sahami, M., Brunskill, E., & Piech, C.
ORCID
Publisher
Center for Open Science
Published
2024-04-25
DOI
10.31219/osf.io/qy8zd
OpenAlex
W4395445411
Cited by
19
References
0
Citations by year
2026: 6 · 2025: 9 · 2024: 4
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Social Sciences › Education › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
Keywords
large language models, ChatGPT, GPT-4, programming education, introductory programming, student engagement, exam performance, student participation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review