Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language
Open accessThe DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
GitHub Copilot is an artificial intelligence tool for automatically generating source code from natural language problem descriptions. Since June 2022, Copilot has officially been available for free to all students as a plug-in to development environments like Visual Studio Code. Prior work exploring OpenAI Codex, the underlying model that powers Copilot, has shown it performs well on typical CS1 problems thus raising concerns about its potential impact on how introductory programming courses are taught. However, little is known about the types of problems for which Copilot does not perform well, or about the natural language interactions that a student might have with Copilot when resolving errors. We explore these questions by evaluating the performance of Copilot on a publicly available dataset of 166 programming problems. We find that it successfully solves around half of these problems on its very first attempt, and that it solves 60% of the remaining problems using only natural language changes to the problem description. We argue that this type of prompt engineering, which we believe will become a standard interaction between human and Copilot when it initially fails, is a potentially useful learning activity that promotes computational thinking skills, and is likely to change the nature of code writing skill development.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This has been inspired by many other instances where LLMs have already demonstrated remarkable effectiveness in generating novel language artifacts. These include course readings [22], code explanations [23, 28, 29, 41], model solutions [9, 35, 42], feedback [34], or responses to help requests [18, 26, 46, 47].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [9]
In the citing paper
“The phrasing of the prompt and problem statement can significantly impact whether the LLM will solve it correctly [Denny et al. 2023].”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Denny et al. 2023
In the citing paper
“Our examination of the prompts indicated that this discrepancy may arise because students face challenges in constructing effective queries or revising them to obtain useful information, aligning with findings in the literature on code generation using LLMs [13].”
From Using an LLM to Help With Code Understanding (2024) · cites this paper as [13]
In the citing paper
“Hence, the findings reported in [46] appear to be challenged by the more capable GPT-4 model as well as the findings reported in [7, 10, 11, 38].”
From Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses (2023) · cites this paper as [7]
In the citing paper
“Hence, the strengths and limitations reported in prior work [7, 10, 11, 38, 46] still hold to a certain degree.”
From Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses (2023) · cites this paper as [7]
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1×
“While their ability to generate novel human-like outputs is on the one hand fascinating, their rapidly increasing deployment has caused alarm among some researchers and led to calls for better understanding of their implications and risks [3, 24].”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · cited 1×
“Indeed, it is well known that language model outputs are very sensitive to their inputs [19].”
BackgroundVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · cited 1×
“applied Codex to the task of generating novel programming exercises given a single example as input [20].”
MethodSelf-citationThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · cited 3×
“Nevertheless, across the 166 problems we evaluated, overall we observe similar rates of initial success reported in earlier studies [11].”
BackgroundResultSelf-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417
Research works printed but not linked: 4
- Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021). · arXiv
- 10.5555/3495724.3495883 · doi:10.5555/3495724.3495883
- Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 (2021). · arXiv
- Leonard Tang, Elizabeth Ke, Nikhil Singh, Bo Feng, Derek Austin, Nakul Verma, and Iddo Drori. 2022. Solving Probability and Statistics Problems by Probabilistic Program Synthesis at Human Level and Predicting Solvability. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners' and Doctoral Consortium,, Maria Mercedes Rodrigo, Noburu Matsuda, Alexandra I. Cristea, and Vania Dimitrova (Eds.). Springer International Publishing, Cham, 612--615.
Cited by
Items 101–200 of 398
Feng, H., Luxton-Reilly, A., Wuensche, B. C., & Denny, P. C. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716658 · Cited by 46
Self-citationAlways Provide Context: The Effects of Code Context on Programming Error Message Enhancement
Santos, E. A., Prasad, P., & Becker, B. A. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617909 · Cited by 36
Will Code Remain a Relevant User Interface for End-User Programming with Generative AI Models?
Sarkar, A. (2023) · Proceedings of the 2023 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3622758.3622882 · Cited by 23
Methodology for Code Synthesis Evaluation of LLMs Presented by a Case Study of ChatGPT and Copilot
Ságodi, Z., Siket, I., & Ferenć, R. (2024) · IEEE Access · doi:10.1109/access.2024.3403858 · Cited by 31
Six Opportunities for Scientists and Engineers to Learn Programming Using AI Tools Such as ChatGPT
Guo, P. J. (2023) · Computing in Science & Engineering · doi:10.1109/mcse.2023.3308476 · Cited by 34
Smith, D. H., Denny, P. C., & Fowler, M. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3662039 · Cited by 40
Self-citationJoshi, I., Budhiraja, R., Tanna, P. D., Jain, L., Deshpande, M., Srivastava, A., Rallapalli, S., Akolekar, H. D., Challa, J. S., & Kumar, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.10694 · Cited by 38
Large language model evaluation for high‐performance computing software development
Godoy, W. F., Valero‐Lara, P., Teranishi, K., Balaprakash, P., & Vetter, J. S. (2024) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.8269 · Cited by 41
LLM4CVE: Enabling Iterative Automated Vulnerability Repair with Large Language Models
Fakih, M., Dharmaji, R., Bouzidi, H., Araya, G. Q., Ogundare, O., Siddika, M., & Faruque, M. A. A. (2025) · Euromicro Conference on Digital System Design (DSD) · doi:10.1109/dsd67783.2025.00087 · Cited by 33
Zhao, G., Yang, L., Hu, B., & Wang, J. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251336154 · Cited by 27
Systematic Literature Review of Prompt Engineering Patterns in Software Engineering
Sasaki, Y., Washizaki, H., Li, J., Sander, D., Yoshioka, N., & Fukazawa, Y. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00096 · Cited by 37
Leveraging LLM Tutoring Systems for Non-Native English Speakers in Introductory CS Courses
Molina, I. V., Montalvo, A., Ochoa, B., Denny, P. C., & Porter, L. (2025) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--56928 · Cited by 28
Self-citationLauren, P., & Watta, P. B. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342970 · Cited by 28
LLM-aided explanations of EDA synthesis errors
Qiu, S., Tan, B., & Pearce, H. A. (2024) · IEEE LLM Aided Design Workshop (LAD) · doi:10.1109/lad62341.2024.10691721 · Cited by 36
Can ChatGPT Play the Role of a Teaching Assistant in an Introductory Programming Course?Preprint
Anishka, Mehta, A., Gupta, N., Balachandran, A., Kumar, D., & Jalote, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.07343 · Cited by 33
Integrating Natural Language Prompting Tasks in Introductory Programming Courses
Kerslake, C. P., Denny, P. C., Smith, D. H., Prather, J. E., Leinonen, J., Luxton-Reilly, A., & MacNeil, S. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690125 · Cited by 33
Self-citationGenerating Programs Trivially: Student Use of Large Language Models
Prasad, S., Greenman, B., Nelson, T., & Krishnamurthi, S. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617921 · Cited by 35
Analyzing Prompt Influence on Automated Method Generation: An Empirical Study with Copilot
Fagadau, I. D., Mariani, L., Micucci, D., & Riganelli, O. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644409 · Cited by 38
Sen, R., & Deng, X. (2025) · Journal of Information Systems Education · doi:10.62273/zlum4022 · Cited by 40
A comparative study of AI ‐generated and human‐crafted learning objectives in computing education
Doyle, A., Sridhar, P., Agarwal, A., Šavelka, J., & Sakr, M. F. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13092 · Cited by 30
Yu, Z., Liu, S., Denny, P. C., Bergen, A., & Liut, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701844 · Cited by 35
Self-citationLópez‐Pernas, S., Misiejuk, K., de Oliveira, E. A., & Saqr, M. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770043 · Cited by 30
Smith, D. H., & Zilles, C. B. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653582 · Cited by 37
Pădurean, V.-A., Denny, P. C., Gotovos, A., & Singla, A. K. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729094 · Cited by 25
Self-citationDeng, L., & Jamaludin, K. A. (2026) · SAGE Open · doi:10.1177/21582440261418315 · Cited by 29
Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions
Lehtinen, T., Koutcheme, C., & Hellas, A. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640058 · Cited by 32
The Role of AI in Transforming Assessment Practices in Education
Vetrivel, S. C., Vidhyapriya, P., & Arun, V. P. (2024) · Advances in educational marketing, administration, and leadership book series · doi:10.4018/979-8-3693-5443-8.ch003 · Cited by 42
Bhattacharjee, A., Xu, S. Y., Rao, P., Zeng, Y., Meyerhoff, J., Ahmed, S. I., Mohr, D. C., Liut, M., Mariakakis, A. T., Kornfield, R., et al. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735810 · Cited by 27
Plaatjies, B. O., & van Wyk, M. M. (2025) · Journal of Teaching and Learning · doi:10.22329/jtl.v19i4.10017 · Cited by 26
Penney, J., Pimentel, J. F., Steinmacher, I., & Gerosa, M. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-54975-5_12 · Cited by 29
Computer Programming Education in the Age of Generative AI: Insights from Empirical ResearchPreprint
Deriba, F. G., Sanusi, I. T., Campbell, O. O., & Oyelere, S. S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4891302 · Cited by 28
Memolet: Reifying the Reuse of User-AI Conversational Memories
Yen, R., & Zhao, J. (2024) · Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3654777.3676388 · Cited by 22
Lee, U., Kim, Y., Lee, S., Park, J., Mun, J., Lee, E., Kim, H., Lim, C., & Yoo, Y. J. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00448-4 · Cited by 30
Is LLM-Generated Code More Maintainable & Reliable Than Human-Written Code?
Molison, A. S., Moraes, M. C., Melo, G., Santos, F., & Assunção, W. K. G. (2025) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem64174.2025.00036 · Cited by 38
Interactions with Prompt Problems: A New Way to Teach Programming with Large Language ModelsPreprint
Prather, J. E., Denny, P. C., Leinonen, J., Smith, D. H., Reeves, B. N., MacNeil, S., Becker, B. A., Luxton-Reilly, A., Amarouche, T., & Kimmel, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.10759 · Cited by 26
Self-citationPrather, J., Reeves, B. N., Denny, P., Leinonen, J., MacNeil, S., Luxton-Reilly, A., Orvalho, J., Alipour, A., Alfageeh, A., Amarouche, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.12800 · Cited by 20
Self-citationGenerative AI in Computer Science Education: Challenges and Opportunities
Franklin, D., Denny, P., Gonzalez-Maldonado, D. A., & Tran, M. (2025) · doi:10.1017/9781009581738 · Cited by 20
Self-citationAgarwal, V., Garg, M. K., Dharmavaram, S., & Kumar, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.01687 · Cited by 27
Comparing Feedback from Large Language Models and Instructors: Teaching Computer Science at Scale
Nguyen, H., Stott, N., & Allan, V. H. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3664660 · Cited by 23
Towards LLM-Assisted System Testing for Microservices
Almutawa, M., Ghabrah, Q., & Canini, M. (2024) · IEEE International Conference on Distributed Computing Systems Workshops (ICDCSW) · doi:10.1109/icdcsw63686.2024.00011 · Cited by 27
A High-Quality Generation Approach for Educational Programming Projects Using LLM
Song, T., Zhang, H., & Xiao, Y. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3499751 · Cited by 21
Automating Personalized Parsons Problems with Customized Contexts and Concepts
Gutierrez, A. D. C., Denny, P. C., & Luxton-Reilly, A. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653568 · Cited by 21
Self-citationCommit Messages in the Age of Large Language ModelsPreprint
Lopes, C. V., Klotzman, V., Ma, I., & Ahmed, I. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.17622 · Cited by 22
Shaping the future of education: a cluster analysis of generative AI’s transformative impact
Issa, H.-E. A., & Maheshwari, G. (2025) · Interactive Technology and Smart Education · doi:10.1108/itse-01-2025-0013 · Cited by 19
Prompt Engineering Guidelines for Using Large Language Models in Requirements Engineering
Ronanki, K., Arvidsson, S., & Axell, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04207-1_17 · Cited by 30
The Social Construction of Generative AI Prompts
Goloujeh, A. M., Sullivan, A., & Magerko, B. S. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650947 · Cited by 24
A Bibliometric Exposition and Review on Leveraging LLMs for Programming Education
Amos, J. P., Amodu, O. A., Mahmood, R. A. R., AbdulQudus, A. B., Zakaria, A. F., Ìyàndá, A., Bukar, U., & Hanapi, Z. (2025) · IEEE Access · doi:10.1109/access.2025.3554627 · Cited by 13
A Brief Summary of Prompting in Using GPT ModelsPreprint
Wang, S., & Jin, P. (2023) · Qeios · doi:10.32388/imzi2q · Cited by 19
Feng, T. H., Denny, P., Wuensche, B., Luxton-Reilly, A., & Hooper, S. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636263 · Cited by 16
Self-citationSantos, P., Cantalejo, K. U., & Moreno, V. (2024) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro60963.2024.10569779 · Cited by 17
Generative AI and CS Education
Johnson, M. (2024) · Communications of the ACM · doi:10.1145/3632523 · Cited by 17
Hu, J., Li, J., Zeng, Y., Yang, D., Liang, D., Meng, H. M. L., & Ma, X. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642960 · Cited by 20
Compiler-Integrated, Conversational AI for Debugging CS1 Programs
Renzella, J., Vassar, A., Solano, L. L., & Taylor, A. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701827 · Cited by 16
Gutierrez, S., Hou, I., Lee, J., Angelikas, K., Man, O., Mettille, S., Prather, J. E., Denny, P. C., & MacNeil, S. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716654 · Cited by 15
Self-citationAnalysis of Generative AI Policies in Computing Course Syllabi
Ali, A., Collier, A. H., Dewan, U., McDonald, N., & Johri, A. (2025) · Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3641554.3701823 · Cited by 14
Solving Proof Block Problems Using Large Language Models
Poulsen, S., Sarsa, S., Prather, J. E., Leinonen, J., Becker, B. A., Hellas, A., Denny, P. C., & Reeves, B. N. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630928 · Cited by 21
Self-citationKohn, T. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70043 · Cited by 15
Utilizing ChatGPT in a Data Structures and Algorithms Course: A Teaching Assistant's Perspective
Jamie, P., Hajihashemi, R., & Alipour, S. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720291 · Cited by 20
Fostering Responsible AI Use Through Negative Expertise: A Contextualized Autocompletion Quiz
MacNeil, S., Prather, J. E., Nabid, R. A., Gutierrez, S., Carvalho, S., Shrestha, S., Denny, P. C., Reeves, B. N., Leinonen, J., & Rossetti, R. L. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729067 · Cited by 19
Self-citationDockerfile Flakiness: Characterization and Repair
Shabani, T., Nashid, N., Alian, P., & Mesbah, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00238 · Cited by 17
Chen, N., Qiu, L. K., Wang, A. Z., Wang, Z., & Yang, Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790726 · Cited by 17
Exploring the Difficulty of Faded Parsons Problems for Programming Education
Fromont, F., Jayamanne, H., & Denny, P. C. (2023) · Australasian Computing Education Conference · doi:10.1145/3576123.3576136 · Cited by 17
Self-citationReš, J., Homoliak, I., Perešíni, M., Smrčka, A., Malinka, K., & Hanáček, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.12671 · Cited by 18
Supervised Fine-Tuning LLMs to Behave as Pedagogical Agents in Programming EducationPreprint
Ross, E., Kansal, Y., Renzella, J., Vassar, A., & Taylor, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.20527 · Cited by 21
AI-Generated Code Not Considered Harmful
Kendon, T., Wu, L., & Aycock, J. (2023) · Proceedings of the 25th Western Canadian Conference on Computing Education · doi:10.1145/3593342.3593349 · Cited by 13
Mailach, A., Gorgosch, D., Siegmund, N., & Siegmund, J. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10561-6 · Cited by 13
“Denny and others found that LLMs can solve most of the typical CS1 problems, and a smaller part with some minor modification to the prompts (Denny et al. 2023).”
InfluentialExplain in Plain Language Questions with Indic Languages: Drawbacks, Affordances, and Opportunities
Smith, D. H., Kumar, V., & Denny, P. C. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-84391-4_1 · Cited by 16
Self-citationIndustrial Experience Report on AI-Assisted Coding in Professional Software Development
Ramler, R., Moser, M., Fischer, L., Nissl, M., & Heinzl, R. (2024) · 2024 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3643795.3648377 · Cited by 12
Šavelka, J., Agarwal, A., Bogart, C., & Sakr, M. F. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-53656-4_8 · Cited by 19
Unveiling the Role of GPT‐4 in Solving LeetCode Programming Problems
Vishnu, S., Sahil, & Garg, N. (2025) · Computer Applications in Engineering Education · doi:10.1002/cae.22815 · Cited by 12
Li, N., Broner, S., Kim, Y., Mizuo, K., Sauder, E., To, C., Wang, A., Gila, O., & Shindler, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701946 · Cited by 15
SocraticAI: Transforming LLMs into Guided CS Tutors Through Scaffolded InteractionPreprint
Sunil, K., & Thakkar, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.03501 · Cited by 15
Feng, H., Denny, P. C., Wuensche, B. C., Luxton-Reilly, A., & Whalley, J. L. (2024) · doi:10.1145/3680533.3697064 · Cited by 15
Self-citationSong, J., Ashktorab, Z., Pan, Q., Dugan, C., Geyer, W., & Malone, T. W. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757486 · Cited by 13
Large Language Models (LLMs) in Programming Learning: The Current Research State and Agenda
Fu, Q., Zhao, Y., Jia, Z., & Zheng, Y. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3622043 · Cited by 19
Towards Characterizing Trust in Generative Artificial Intelligence among Students
Amoozadeh, M., Daniels, D., Chen, S., Nam, D., Kumar, A., Hilton, M. L., Alipour, M. A., & Ragavan, S. S. (2023) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3568812.3603469 · Cited by 11
Generative AI in CS Education: Literature Review through a SWOT Lens
Roberts, J., & Mohamed, A. (2024) · Western Canadian Conference on Computing Education · doi:10.1145/3660650.3660657 · Cited by 14
Probeable Problems for Beginner-level Programming-with-AI Contests
Pawagi, M., & Kumar, V. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671108 · Cited by 11
Self-citationExplaining GitHub Actions Failures with Large Language Models: Challenges, Insights, and Limitations
Valenzuela-Toledo, P., Wu, C., Hernández, S., Boll, A., Machacek, R., Panichella, S., & Kehrer, T. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00037 · Cited by 11
Gambo, O. O., Onifade, O. J., Olanrewaju-Smart, W., Abuya, O. J., Salawu, R. E., Massenon, R., & Gambo, I. (2026) · Discover Education · doi:10.1007/s44217-026-01142-4 · Cited by 11
Prompt Engineering in Medical Image Segmentation: An Overview of the Paradigm Shift
Ali, H., Bulbul, M. F., & Shah, Z. (2023) · International Conference on Artificial Intelligence, Blockchain, and Internet of Things (AIBThings) · doi:10.1109/aibthings58340.2023.10292475 · Cited by 12
Song, T., Tian, Q., Xiao, Y., & Liu, S. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-0730-0_28 · Cited by 15
Assessing Live Programming for Program Comprehension
Graf, O., Thorgeirsson, S., & Su, Z. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653547 · Cited by 10
Cipriano, B. P., Alves, P., & Denny, P. C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72315-5_6 · Cited by 14
Self-citationPwnPilot: Reflections on Trusting Trust in the Age of Large Language Models and AI Code Assistants
Horne, D. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00396 · Cited by 15
Sankaranarayanan, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.20206 · Cited by 14
The Robots are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00658 · Cited by 9
Self-citationAlnahdi, A., & Narain, S. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00020 · Cited by 12
Generative Trigger-Action Programming with Ply
Aveni, T. J., Mor, H., Fox, A., & Hartmann, B. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747638 · Cited by 12
The Impact of AI Coding Assistants on Software Engineering: A Longitudinal Study
Vella, A., & Blincoe, K. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3861343 · Cited by 14
A Bug's New Life: Creating Refute Questions from Filtered CS1 Student Code Snapshots
Agarwal, N., Kumar, V., Raman, A., & Karkare, A. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617916 · Cited by 10
Self-citationBeyond Automation: AI-Driven Project Management with OpenAI and Prompt Engineering
Jayaram, M., Bhutkar, Y., Bojjanapalli, I. L. K., Yeshwanth, G., & Reddy, B. Y. (2024) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet61485.2024.10698333 · Cited by 12
Visualisation Design Ideation with AI: A New Framework, Vocabulary, and Tool
Owen, A. E., & Roberts, J. C. (2024) · Future Internet · doi:10.3390/fi16110406 · Cited by 9
Martínez, J., Llinás, B., Botello, J. G., Padilla, J. J., & Frydenlund, E. F. (2024) · Winter Simulation Conference · doi:10.1109/wsc63780.2024.10838967 · Cited by 15
Stone, I. (2023) · Conference on Human Centred Artificial Intelligence - Education and Practice (HCAIep) · doi:10.1145/3633083.3633099 · Cited by 9
Dcc --help: Generating Context-Aware Compiler Error Explanations with Large Language ModelsPreprint
Taylor, A., Vassar, A., Renzella, J., & Pearce, H. A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.11873 · Cited by 10
Alario-Hoyos, C., Kemcha, R., Kloos, C. D., Callejo, P., Estévez-Ayres, I., Santín-Cristóbal, D., Cruz-Argudo, F., & López-Sánchez, J. L. (2024) · 2024 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE) · doi:10.1109/tale62452.2024.10834365 · Cited by 8
The underlying potential of NLP for microcontroller programming education
Rocha, A., Sousa, L., Alves, M., & Sousa, A. J. (2024) · Computer Applications in Engineering Education · doi:10.1002/cae.22778 · Cited by 8
Okopnyi, P., Nordberg, O. E., & Guribye, F. (2024) · Halfway to the Future Symposium · doi:10.1145/3686169.3686184 · Cited by 8
Metadata
- Authors
- Denny, P., Kumar, V., & Giacaman, N.
- Affiliations
- University of Auckland (NZ)
- Indian Institute of Science Bangalore (IN)
- Venue
- Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2023-03-02
- Pages
- 1136-1142
- OpenAlex
- W4323037544
- Cited by
- 287
- References
- 22
- Citations by year
- 2026: 57 · 2025: 84 · 2024: 99 · 2023: 47
- FWCI
- 86.69
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- GitHub Copilot, prompt engineering, natural language programming, OpenAI Codex, code generation, introductory programming education, computational thinking, error resolution
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.