Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language
Open accessThe DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
GitHub Copilot is an artificial intelligence tool for automatically generating source code from natural language problem descriptions. Since June 2022, Copilot has officially been available for free to all students as a plug-in to development environments like Visual Studio Code. Prior work exploring OpenAI Codex, the underlying model that powers Copilot, has shown it performs well on typical CS1 problems thus raising concerns about its potential impact on how introductory programming courses are taught. However, little is known about the types of problems for which Copilot does not perform well, or about the natural language interactions that a student might have with Copilot when resolving errors. We explore these questions by evaluating the performance of Copilot on a publicly available dataset of 166 programming problems. We find that it successfully solves around half of these problems on its very first attempt, and that it solves 60% of the remaining problems using only natural language changes to the problem description. We argue that this type of prompt engineering, which we believe will become a standard interaction between human and Copilot when it initially fails, is a potentially useful learning activity that promotes computational thinking skills, and is likely to change the nature of code writing skill development.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This has been inspired by many other instances where LLMs have already demonstrated remarkable effectiveness in generating novel language artifacts. These include course readings [22], code explanations [23, 28, 29, 41], model solutions [9, 35, 42], feedback [34], or responses to help requests [18, 26, 46, 47].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [9]
In the citing paper
“The phrasing of the prompt and problem statement can significantly impact whether the LLM will solve it correctly [Denny et al. 2023].”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Denny et al. 2023
In the citing paper
“Our examination of the prompts indicated that this discrepancy may arise because students face challenges in constructing effective queries or revising them to obtain useful information, aligning with findings in the literature on code generation using LLMs [13].”
From Using an LLM to Help With Code Understanding (2024) · cites this paper as [13]
In the citing paper
“Hence, the findings reported in [46] appear to be challenged by the more capable GPT-4 model as well as the findings reported in [7, 10, 11, 38].”
From Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses (2023) · cites this paper as [7]
In the citing paper
“Hence, the strengths and limitations reported in prior work [7, 10, 11, 38, 46] still hold to a certain degree.”
From Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses (2023) · cites this paper as [7]
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“While their ability to generate novel human-like outputs is on the one hand fascinating, their rapidly increasing deployment has caused alarm among some researchers and led to calls for better understanding of their implications and risks [3, 24].”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · cited 1×
“Indeed, it is well known that language model outputs are very sensitive to their inputs [19].”
BackgroundVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · cited 1×
“applied Codex to the task of generating novel programming exercises given a single example as input [20].”
MethodSelf-citationThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · cited 3×
“Nevertheless, across the 166 problems we evaluated, overall we observe similar rates of initial success reported in earlier studies [11].”
BackgroundResultSelf-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417
Research works printed but not linked: 4
- Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021). · arXiv
- 10.5555/3495724.3495883 · doi:10.5555/3495724.3495883
- Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 (2021). · arXiv
- Leonard Tang, Elizabeth Ke, Nikhil Singh, Bo Feng, Derek Austin, Nakul Verma, and Iddo Drori. 2022. Solving Probability and Statistics Problems by Probabilistic Program Synthesis at Human Level and Predicting Solvability. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners' and Doctoral Consortium,, Maria Mercedes Rodrigo, Noburu Matsuda, Alexandra I. Cristea, and Vania Dimitrova (Eds.). Springer International Publishing, Cham, 612--615.
Cited by
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784
“Denny et al. (2023) [18] Investigates Copilot, an AI-powered code generation tool, emphasizing its problem-solving capabilities and discussing prompt engineering as a means to enhance computational thinking skills.”
BackgroundLarge language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“Denny et al. [60] argued that prompt engineering is a useful learning activity that fosters software engineering students’ computational thinking.”
BackgroundLarge Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
Self-citationUsing an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
Computing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“Thus, AI-generated solutions can be a time-efficient option [11].”
BackgroundInfluentialSelf-citationKazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Denny, P., Craig, M., & Grossman, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642773 · Cited by 290
Self-citationLau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241
Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · arXiv · doi:10.48550/arxiv.2306.05715 · Cited by 211
Comparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588785 · Cited by 210
Self-citationA Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2023) · arXiv · doi:10.48550/arxiv.2303.17125 · Cited by 206
A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
Liang, J. T., Yang, C., & Myers, B. A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3608128 · Cited by 203
Kazemitabaar, M., Hou, X., Henley, A. Z., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631806 · Cited by 276
CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192
Self-citationPrompt Problems: A New Programming Exercise for the Generative AI Era
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630909 · Cited by 187
Self-citationLarge Language Models in Computer Science Education: A Systematic Literature Review
Raihan, M. N., Siddiq, M. L., Santos, J. C. S., & Zampieri, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701863 · Cited by 219
On Hardware Security Bug Code Fixes by Prompting Large Language Models
Ahmad, B., Thakur, S., Tan, B., Karri, R., & Pearce, H. A. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3374558 · Cited by 196
Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145
Getting pwn’d by AI: Penetration Testing with Large Language Models
Happe, A., & Cito, J. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3613083 · Cited by 144
“Prompts have to be carefully prepared, yielding a new discipline that has been called prompt engineering [9, 18, 32, 35].”
BackgroundThe Impact of Large Language Models on Programming Education and Student Learning Outcomes
Jošt, G., Taneski, V., & Karakatič, S. (2024) · Applied Sciences · doi:10.3390/app14104115 · Cited by 143
Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141
“the strengths and limitations reported in prior work [7, 10, 11, 38, 46] still hold to a certain degree.”
BackgroundMethodResultInfluentialGenerative AI in Computing Education: Perspectives of Students and Instructors
Zastudil, C., Rogalska, M., Kapp, C., Vaughn, J., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343467 · Cited by 136
“Researchers are already demonstrating the capabilities of these models to help students generate code [2], [3], [25], explain code to students [5], [6], [9], and even produce programming assignments [1], [4].”
BackgroundDesirable Characteristics for AI Teaching Assistants in Programming Education
Denny, P. C., MacNeil, S., Šavelka, J., Porter, L., & Luxton-Reilly, A. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653574 · Cited by 153
Self-citationA Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132
AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744217 · Cited by 163
Evaluating LLM-generated Worked Examples in an Introductory Programming Course
Jury, B., Lorusso, A., Leinonen, J., Denny, P. C., & Luxton-Reilly, A. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636252 · Cited by 145
Self-citationIs It AI or Is It Me? Understanding Users’ Prompt Journey with Text-to-Image Generative AI Tools
Goloujeh, A. M., Sullivan, A., & Magerko, B. S. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642861 · Cited by 156
Exploring the Potential of Large Language Models to Generate Formative Programming Feedback
Kiesler, N., Lohr, D., & Keuning, H. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343457 · Cited by 141
Savelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588792 · Cited by 109
“[10] focused on the effects of prompt engineering when applying Copilot to a set of 166 exercises from the publicly available CodeCheck repository.”
MethodPatterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant
Sheese, B., Liffiton, M., Savelka, J., & Denny, P. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636249 · Cited by 74
Self-citationWhat is wrong with your code generated by large language models? An extensive study
Dou, S., Jia, H., Wu, S., Zheng, H., Wu, M., Tao, Y., Zhang, M., Chai, M., Fan, J., Xu, N., et al. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4632-8 · Cited by 117
Denny, P. C., Khosravi, H., Hellas, A., Leinonen, J., & Sarsa, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.10509 · Cited by 118
Self-citationHow Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering
Choudhuri, R., Liu, D., Steinmacher, I., Gerosa, M., & Sarma, A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639201 · Cited by 72
The Effects of Generative AI on Computing Students’ Help-Seeking Preferences
Hou, I., Mettille, S., Man, O., Li, Z., Zastudil, C., & MacNeil, S. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636248 · Cited by 97
Next-Step Hint Generation for Introductory Programming Using Large Language Models
Roest, L., Keuning, H., & Jeuring, J. T. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636259 · Cited by 123
Piccolo, S., Denny, P. C., Luxton-Reilly, A., Payne, S., & Ridge, P. G. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011511 · Cited by 104
Self-citationGenerative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factors
Ghimire, A., Pather, J., & Edwards, J. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892891 · Cited by 88
Koutcheme, C., Dainese, N., Sarsa, S., Hellas, A., Leinonen, J., & Denny, P. C. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653612 · Cited by 106
Self-citationMargulieux, L. E., Prather, J., Reeves, B. N., Becker, B. A., Cetin Uzun, G., Loksa, D., Leinonen, J., & Denny, P. (2024) · Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3649217.3653621 · Cited by 59
Self-citationHarnessing LLMs in Curricular Design: Using GPT-4 to Support Authoring of Learning Objectives
Sridhar, P., Doyle, A., Agarwal, A., Bogart, C., Savelka, J., & Sakr, M. (2023) · arXiv · doi:10.48550/arxiv.2306.17459
Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code GeneratorsPreprint
Denny, P. C., Leinonen, J., Prather, J. E., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16364 · Cited by 96
Self-citationWang, T., Vargas-Díaz, D., Brown, C., & Chen, Y. (2023) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00018 · Cited by 97
Automatic Programming: Large Language Models and Beyond
Lyu, M. R.-T., Ray, B., Roychoudhury, A., Tan, S. H., & Thongtanunam, P. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708519 · Cited by 131
Detecting ChatGPT-Generated Code Submissions in a CS1 Course Using Machine Learning Models
Hoq, M., Shi, Y., Leinonen, J., Babalola, D., Lynch, C. F., Price, T., & Akram, B. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630826 · Cited by 96
Incorporating Generative AI into Software Development Education
Petrovska, O., Clift, L., Moller, F., & Pearsall, R. (2024) · Proceedings of the 8th Conference on Computing Education Practice · doi:10.1145/3633053.3633057 · Cited by 75
CodeTailor: LLM-Powered Personalized Parsons Puzzles for Engaging Support While Learning Programming
Hou, X., Wu, Z., Wang, X., & Ericson, B. J. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3662032 · Cited by 90
AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · arXiv · doi:10.48550/arxiv.2503.00079 · Cited by 73
Hsu, H.-P. (2025) · TechTrends · doi:10.1007/s11528-025-01052-6 · Cited by 83
Generative Artificial Intelligence for Software Engineering - a Research Agenda
NguyenDuc, A., Cabrero-Daniel, B., Arora, C., Przybylek, A., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.-K., et al. (2023) · doi:10.2139/ssrn.4622517 · Cited by 71
ChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course
Popovici, M.-D. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2269006 · Cited by 70
“The study of Denny et al. (2023) shifted focus from productivity to evaluating Copilot’s performance on a publicly accessible dataset comprising 166 programming problems.”
InfluentialMany bioinformatics programming tasks can be automated with ChatGPT
Piccolo, S. R., Denny, P., Luxton-Reilly, A., Payne, S., & Ridge, P. G. (2023) · arXiv · doi:10.48550/arxiv.2303.13528 · Cited by 69
Self-citationDenny, P. C., Smith, D. H., Fowler, M., Prather, J. E., Becker, B. A., & Leinonen, J. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653587 · Cited by 86
Self-citationPrasad, P., & Sane, A. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630828 · Cited by 80
Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments
Balse, R., Valaboju, B., Singhal, S., Warriem, J. M., & Prasad, P. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588852 · Cited by 63
Generating Multiple Choice Questions for Computing Courses Using Large Language Models
Tran, A., Angelikas, K., Rama, E., Okechukwu, C., Smith, D. H., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342898 · Cited by 60
“Recently, Large Language Models (LLMs) have demonstrated near human-level performance across a variety of tasks such as generating code [10], code explanations [11], [12], and programming assignments [13], [14].”
BackgroundGenerative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60
Software Engineering by and for Humans in an AI Era
Abrahão, S., Grundy, J., Pezzè, M., Storey, M., & Tamburri, D. A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715111 · Cited by 81
Taylor, A., Vassar, A., Renzella, J., & Pearce, H. A. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630822 · Cited by 68
Computing education using generative artificial intelligence tools: A systematic literature review
Agbo, F. J., Olivia, C., Oguibe, G. F., Sanusi, I. T., & Sani, G. (2025) · Computers and Education Open · doi:10.1016/j.caeo.2025.100266 · Cited by 75
Bhattacharjee, A., Zeng, Y., Xu, S. Y., Kulzhabayeva, D., Ma, M., Kornfield, R., Ahmed, S. I., Mariakakis, A. T., Czerwinski, M. P., Kuzminykh, A., et al. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642081 · Cited by 73
Reeves, B., Sarsa, S., Prather, J., Denny, P., Becker, B. A., Hellas, A., Kimmel, B., Powell, G., & Leinonen, J. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588805 · Cited by 56
Self-citationPerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedback
Peng, Y. F., Gotmare, A. D., Lyu, M. R.-T., Xiong, C., Savarese, S., & Sahoo, D. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00008 · Cited by 77
Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study
Sheard, J., Denny, P., Hellas, A., Leinonen, J., Malmi, L., & Simon (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630880 · Cited by 54
Self-citationToward General Design Principles for Generative AI Applications
Weisz, J. D., Muller, M., He, J., & Houde, S. (2023) · arXiv · doi:10.48550/arxiv.2301.05578 · Cited by 54
Generative Artificial Intelligence: Fundamentals
Corchado, J. M., Sebastian López, F., V., J., Raul Garcia, S., & Chamoso, P. (2023) · ADCAIJ ADVANCES IN DISTRIBUTED COMPUTING AND ARTIFICIAL INTELLIGENCE JOURNAL · doi:10.14201/adcaij.31704 · Cited by 62
Liu, J., & Li, S. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241240460 · Cited by 62
Beyond Code Generation: LLM-supported Exploration of the Program Design Space
Zamfirescu-Pereira, J. D., Jun, E., Terry, M. A., Yang, Q., & Hartmann, B. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714154 · Cited by 79
Budhiraja, R., Joshi, I., Challa, J. S., Akolekar, H. D., & Kumar, D. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636257 · Cited by 52
Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation
Godoy, W. F., Valero‐Lara, P., Teranishi, K., Balaprakash, P., & Vetter, J. S. (2023) · International Conference on Parallel Processing Workshops (ICPPW) · doi:10.1145/3605731.3605886 · Cited by 66
Using GPT-4 to Provide Tiered, Formative Code Feedback
Nguyen, H., & Allan, V. H. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630960 · Cited by 72
Lohr, D., Keuning, H., & Kiesler, N. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13107 · Cited by 63
Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models
MacNeil, S., Denny, P. C., Tran, A., Leinonen, J., Bernstein, S., Hellas, A., Sarsa, S., & Kim, J. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636245 · Cited by 56
Self-citationAbolnejadian, M., Alipour, S., & Taeb, K. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3637148 · Cited by 65
Garg, A., Nisumba Soodhani, K., & Rajendran, R. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100380 · Cited by 49
An Industry Case Study on Adoption of AI-based Programming Assistants
Davila, N., Wiese, I., Steinmacher, I., da Silva, L. L., Kawamoto, A. L. S., Favaro, G. J. P., & Nunes, I. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3639477.3643648 · Cited by 61
Humble, N. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00494-x · Cited by 57
Akolekar, H. D., Jhamnani, P., Kumar, V., Tailor, V., Pote, A., Meena, A., Kumar, K., Challa, J. S., & Kumar, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-93871-z · Cited by 56
Self-citationMahon, J., Namee, B. M., & Becker, B. A. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653602 · Cited by 57
Combining LLM-Generated and Test-Based Feedback in a MOOC for Programming
Gabbay, H., & Cohen, A. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3662040 · Cited by 68
Hou, I., Man, O., Mettille, S., Gutierrez, S., Angelikas, K., & MacNeil, S. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636247 · Cited by 47
How Do Programming Students Use Generative AI?
Rahe, C., & Maalej, W. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715762 · Cited by 45
“The phrasing of the prompt and problem statement can significantly impact whether the LLM will solve it correctly [Denny et al. 2023].”
From Today’s Code to Tomorrow’s Symphony: The AI Transformation of Developer’s Routine by 2030
Qiu, K., Puccinelli, N., Ciniselli, M., & Grazia, L. D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709353 · Cited by 50
Large language models meet user interfaces: The case of provisioning feedback
Pozdniakov, S., Brazil, J., Abdi, S., Bakharia, A., Sadiq, S., Gašević, D., Denny, P., & Khosravi, H. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100289 · Cited by 43
Self-citationAnalyzing LLM Usage in an Advanced Computing Class in India
Arora, U., Garg, A., Gupta, A., Jain, S., Mehta, R. L., Oberoi, R., Prachi, Raina, A., Saini, M., Sharma, S., et al. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716657 · Cited by 52
Evaluating the Quality of LLM-Generated Explanations for Logical Errors in CS1 Student Programs
Balse, R., Kumar, V., Prasad, P., & Warriem, J. M. (2023) · Bangalore Annual Compute Conference · doi:10.1145/3627217.3627233 · Cited by 55
Self-citationPirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 65
Exploring Student-AI Interactions in Vibe Coding
Geng, F., Shah, A., Li, H., Mulla, N., Swanson, S., Raj, G. S., Zingaro, D., & Porter, L. (2026) · Australasian Computing Education Conference · doi:10.1145/3786228.3786236 · Cited by 51
Implications of ChatGPT for Data Science Education
Shen, Y., Ai, X., Raj, A. G. S., John, R. J. L., & Syamkumar, M. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630874 · Cited by 50
Large Language Models in Computer Science Classrooms: Ethical Challenges and Strategic Solutions
Azoulay, R., Hirst, T., & Reches, S. (2025) · Applied Sciences · doi:10.3390/app15041793 · Cited by 47
Jordan, M., Ly, K., & Soosai Raj, A. G. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630897 · Cited by 39
Chen, B., Lewis, C. M., West, M., & Zilles, C. (2024) · Proceedings of the Eleventh ACM Conference on Learning @ Scale · doi:10.1145/3657604.3662046 · Cited by 39
CS1 with a Side of AI: Teaching Software Verification for Secure Code in the Era of Generative AI
Fernandez, A. S., & Cornell, K. A. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630817 · Cited by 45
DBox: Scaffolding Algorithmic Programming Learning through Learner-LLM Co-Decomposition
Ma, S., Wang, J., Zhang, Y., Ma, X., & Wang, A. Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713748 · Cited by 63
Enhancing Programming Error Messages in Real Time with Generative AI
Kimmel, B., Geisert, A. L., Yaro, L., Gipson, B., Hotchkiss, R., Osae-Asante, S. K., Vaught, H., Wininger, G., & Yamaguchi, C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3647967 · Cited by 44
Integrating AI Tutors in a Programming Course
Ma, I., Krone-Martins, A., & Lopes, C. V. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690094 · Cited by 55
Human-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 49
A New AI Literacy For The Algorithmic Age: Prompt Engineering Or Eductional Promptization?
Haugsbakken, H., & Hagelia, M. (2024) · International Conference on Applied Artificial Intelligence (ICAPAI) · doi:10.1109/icapai61893.2024.10541229 · Cited by 44
Prather, J. E., Reeves, B. N., Denny, P. C., Leinonen, J., MacNeil, S., Luxton-Reilly, A., Orvalho, J. G., Alipour, M. A., Alfageeh, A., Amarouche, T., et al. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716649 · Cited by 44
Self-citation
Metadata
- Authors
- Denny, P., Kumar, V., & Giacaman, N.
- Affiliations
- University of Auckland (NZ)
- Indian Institute of Science Bangalore (IN)
- Venue
- Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2023-03-02
- Pages
- 1136-1142
- OpenAlex
- W4323037544
- Cited by
- 287
- References
- 22
- Citations by year
- 2026: 57 · 2025: 84 · 2024: 99 · 2023: 47
- FWCI
- 86.69
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- GitHub Copilot, prompt engineering, natural language programming, OpenAI Codex, code generation, introductory programming education, computational thinking, error resolution
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.