Does ChatGPT Help With Introductory Programming?An Experiment of Students Using ChatGPT in CS1
Open accessAbstract
Generative AI, notably ChatGPT, has garnered attention in computer science education. This paper presents a controlled experiment that explores ChatGPT's role in CS1 in a classroom setting. Specifically, we aim to investigate the impact of ChatGPT on student learning outcomes and their behaviors when working on programming assignments. Participants were tasked with creating a UML diagram and subsequently implementing its design through programming, followed by a closed-book post-evaluation and a post-survey. All the participants were required to screen-record the whole process. In total, 56 participants were recruited, with 48 successful screen recordings. Participants in the Experimental Group can access ChatGPT 3.5 and other online resources, such as Google and Stack Overflow when creating the UML diagram and programming; however, participants in the Control Group can access all online resources except for ChatGPT (i.e., the only design variable is the access to ChatGPT). Finally, we measured and analyzed participants' learning outcomes through their UML diagram, programming, and post-evaluation scores. We also analyzed the time participants took to complete the tasks and their interactions with ChatGPT and other resources from the screen recordings. After finishing the tasks, student participants also provided their perceptions of using ChatGPT in CS1 through a post-survey.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 17 · Cited by 96
Research works printed but not linked: 21
- David Baidoo-Anu and Leticia Owusu Ansah. 2023. Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Available at SSRN 4337484 (2023).
- Som Biswas. 2023. Role of ChatGPT in Computer Programming.: ChatGPT in Computer Programming. Mesopotamian Journal of Computer Science 2023 (2023), 8-16.
- Philipp Bitzenbauer. 2023. ChatGPT in physics education: A pilot study on easy-to-implement activities. Contemporary Educational Technology 15, 3 (2023), ep430.
- Sebastian Bordt and Ulrike von Luxburg. 2023. Chatgpt participates in a computer science exam. arXiv preprint arXiv:2303.09461 (2023). · arXiv
- E Chen, R Huang, HS Chen, YH Tseng, and LY Li. 2023. GPTutor: A ChatGPT-powered programming tool for code explanation. arXiv. arXiv preprint arXiv:2305.01863 (2023). · arXiv
- Yunhe Feng, Sreecharan Vanam, Manasa Cherukupally, Weijian Zheng, Meikang Qiu, and Haihua Chen. 2023. Investigating Code Generation Performance of Chat- GPT with Crowdsourcing Social Data. In Proceedings of the 47th IEEE Computer Software and Applications Conference. 1-10.
- Mohammad Fraiwan and Natheer Khasawneh. 2023. A Review of ChatGPT Applications in Education, Marketing, Software Engineering, and Healthcare: Benefits, Drawbacks, and Research Directions. arXiv preprint arXiv:2305.00237 (2023). · arXiv
- Natalie Kiesler, Dominic Lohr, and Hieke Keuning. 2023. Exploring the Potential of Large Language Models to Generate Formative Programming Feedback. arXiv preprint arXiv:2309.00029 (2023). · arXiv
- Natalie Kiesler and Daniel Schiffner. 2023. Large Language Models in Intro- ductory Programming Education: ChatGPT's Performance and Implications for Assessments. arXiv preprint arXiv:2308.08572 (2023). · arXiv
- Lucas Kohnke, Benjamin Luke Moorhouse, and Di Zou. 2023. ChatGPT for language teaching and learning. RELC Journal (2023), 00336882231162868.
- Sam Lau and Philip Guo. 2023. From" Ban It Till We Understand It" to" Resistance is Futile": How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools such as ChatGPT and GitHub Copilot. In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1. 106-121.
- Daniel Leiker, Ashley Ricker Gyllen, Ismail Eldesouky, and Mutlu Cukurova. 2023. Generative AI for learning: Investigating the potential of synthetic learning videos. arXiv preprint arXiv:2304.03784 (2023). · arXiv
- Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang. 2022. Generating diverse code explanations using the gpt-3 large language model. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 2. 37-39.
- Zachary A Pardos and Shreya Bhandari. 2023. Learning gain differences be- tween ChatGPT and human tutor generated algebra hints. arXiv preprint arXiv:2302.06871 (2023). · arXiv
- Basit Qureshi. 2023. Exploring the use of chatgpt as a tool for learning and assessment in undergraduate computer science curriculum: Opportunities and challenges. arXiv preprint arXiv:2304.11214 (2023). · arXiv
- Saumyaranjan Sahoo, Satish Kumar, Mohammad Zoynul Abedin, Weng Marc Lim, and Suresh Kumar Jakhar. 2023. Deep learning applications in manufacturing operations: a review of trends and ways forward. Journal of Enterprise Information Management 36, 1 (2023), 221-251.
- Malik Sallam. 2023. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare, Vol. 11. MDPI, 887.
- Xingzhi Wang, Nabil Anwer, Yun Dai, and Ang Liu. 2023. ChatGPT for design, manufacturing, and education. Procedia CIRP 119 (2023), 7-14.
- Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023). · arXiv
- Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. 2023. A brief overview of ChatGPT: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica 10, 5 (2023), 1122-1136.
- Cynthia Zastudil, Magdalena Rogalska, Christine Kapp, Jennifer Vaughn, and Stephen MacNeil. 2023. Generative AI in Computing Education: Perspectives of Students and Instructors. arXiv preprint arXiv:2308.04309 (2023). · arXiv
- Authors
- Xue, Y., Chen, H., Bai, G. R., Tairas, R., & Huang, Y.
- Affiliations
- Vanderbilt University (US)
- Venue
- Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training
- Publisher
- ACM
- Published
- 2024-04-14
- Pages
- 331-341
- OpenAlex
- W4398785929
- Cited by
- 96
- References
- 17
- Citations by year
- 2026: 40 · 2025: 48 · 2024: 8
- FWCI
- 42.11
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) (publishedVersion)
- Funders
- NSF (CCF-2211429)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- ChatGPT, generative AI in education, introductory programming, CS1, UML diagrams, learning outcomes, student behavior, screen recording, student perceptions, programming exercises, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
