Using GitHub Copilot to Solve Simple Programming Problems
Open accessAbstract
The teaching and assessment of introductory programming involves writing code that solves a problem described by text. Previous research found that OpenAI's Codex, a natural language machine learning model trained on billions of lines of code, performs well on many programming problems, often generating correct and readable Python code. GitHub's version of Codex, Copilot, is freely available to students. This raises pedagogic and academic integrity concerns. Educators need to know what Copilot is capable of, in order to adapt their teaching to AI-powered programming assistants. Previous research evaluated the most performant Codex model quantitatively, e.g. how many problems have at least one correct suggestion that passes all tests. Here I evaluate Copilot instead, to see if and how it differs from Codex, and look qualitatively at the generated suggestions, to understand the limitations of Copilot. I also report on the experience of using Copilot for other activities asked of students in programming courses: explaining code, generating tests and fixing bugs. The paper concludes with a discussion of the implications of the observed capabilities for the teaching of programming.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Concerns regarding academic honesty and student learning have dominated discourse on GenAI in higher education.
“Given the ability of GenAI tools to generate human-like content, such as language and code, concerns regarding academic honesty and student learning have dominated discourse [16, 29, 33, 36, 55, 115, 117].”
From "I'd Never Actually Realized How Big An Impact It Had Until Now": Perspectives of University Students with Disabilities on Generative Artificial Intelligence (2025) · cites this paper as [115]
The Abstract says that students’ access to Copilot raises pedagogic and academic-integrity concerns, and §7 (Concluding Remarks; page unavailable) discusses implications for student learning. The paper addresses introductory programming, however, and does not establish that these concerns have dominated discourse about GenAI across higher education. (LLM, conf=0.85)
Experimental — accuracy not yet measuredStudents may spend excessive time on prompt engineering in the hope of hitting on a successful solution rather than making progress towards the solution, as observed in the 'shepherding' interaction pattern.
“Wermelinger [191] speculated that students may spend excessive time on prompt engineering in the hope of hitting on a successful solution rather than making progress towards the solution, which was indeed observed in the study by Prather et al. [159] in the 'shepherding' interaction pattern.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
Automatically generated tests may be partially complete, leading students to inadequately test their code.
“Automatically generated tests may be partially complete, leading students to inadequately test their code [191].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
Explanations generated by Copilot focused on a line-by-line description of what the code did rather than how it achieved the goal desired.
“Wermelinger [191] observed that explanations generated by Copilot focused on a line-by-line description of what the code did rather than how it achieved the goal desired.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
Solutions generated by AI might not be easily identified as AI-generated, which requires teachers to adapt and develop new teaching strategies to ensure academic integrity is maintained.
“The solutions generated by AI might not be easily identified as AI-generated [160], which requires teachers to adapt and develop new teaching strategies to ensure academic integrity is maintained [191].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
Students still need to dedicate time to learning algorithmic thinking, program comprehension, debugging, and communication skills in order to become proficient computing experts and to use LLMs effectively.
“Although LLM-based tools like Copilot may serve as a useful springboard for solving CS1 problems, students still need to dedicate time to learning algorithmic thinking, program comprehension, debugging, and communication skills [191] in order to become not only proficient computing experts but also to use LLMs effectively.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
The less variation in Copilot completions might be due to the use of a relatively high temperature value for GPT-4 and GPT-3.5, while Copilot likely uses a lower temperature value.
“This might be due to the use of a relatively high temperature value for these models-previous work found similar results and speculated that Copilot likely uses a lower temperature value [191].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Learning to program = learning to construct mechanisms and explanations
Soloway, E. (1986) · CACM · doi:10.1145/6592.6594 · Cited by 615 · see citing works
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · see citing works
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · see citing works
Novice programmer errors: language constructs and plan composition
Ebrahimi, A. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1069 · Cited by 155 · see citing works
Lobb, R., & Harlow, J. (2016) · ACM Inroads · doi:10.1145/2810041 · Cited by 120 · see citing works
Pattern-oriented instruction and its influence on problem decomposition and solution construction
Muller, O., Ginat, D., & Haberman, B. (2007) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1268784.1268830 · Cited by 113 · see citing works
The recurring rainfall problem
Fisler, K. (2014) · doi:10.1145/2632320.2632346 · Cited by 112 · see citing works
Soloway's Rainfall Problem Has Become Harder
Simon (2013) · doi:10.1109/latice.2013.44 · Cited by 61 · see citing works
Revisiting rainfall to explore exam questions and performance on CS1
Lakanen, A.-J., Lappalainen, V., & Isomöttönen, V. (2015) · doi:10.1145/2828959.2828970 · Cited by 17 · see citing works
Piwek, P., Michel Wermelinger, B., Laney, R., & Walker, R. D. (2019) · Conference on Computing Education Practice (CEP) · doi:10.1145/3294016.3294024 · Cited by 10 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784 · see citing works
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417 · see citing works
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · see citing works
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256 · see citing works
“Wermelinger [106] evaluated the capabilities of Copilot in the spirit of Finnie-Ainsley et al. [42] who evaluated Codex and found it to still miss the mark more often than not.”
MethodLau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241 · see citing works
Navigating the Complexity of Generative AI Adoption in Software Engineering
Russo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652154 · Cited by 217 · see citing works
“While Wermelinger explored the implications of Copilot on teaching and assessment methods in programming courses [97], Dakhel et al.”
BackgroundExploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145 · see citing works
The Impact of Large Language Models on Programming Education and Student Learning Outcomes
Jošt, G., Taneski, V., & Karakatič, S. (2024) · Applied Sciences · doi:10.3390/app14104115 · Cited by 143 · see citing works
Generative AI in Computing Education: Perspectives of Students and Instructors
Zastudil, C., Rogalska, M., Kapp, C., Vaughn, J., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343467 · Cited by 136 · see citing works
Savelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588792 · Cited by 109 · see citing works
Bull, C., & Kharrufa, A. (2024) · IEEE Software · doi:10.1109/ms.2023.3300574 · Cited by 82 · see citing works
Building machines that learn and think with people
Collins, K. M., Sucholutsky, I., Bhatt, U., Chandra, K., Wong, L., Lee, M., Zhang, C. E., Zhi‐Xuan, T., Ho, M. K., Mansinghka, V. K., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01991-9 · Cited by 93 · see citing works
ChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course
Popovici, M.-D. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2269006 · Cited by 70 · see citing works
Transitioning from MLOps to LLMOps: Navigating the Unique Challenges of Large Language Models
Pahune, S. A., & Akhtar, Z. (2025) · Information · doi:10.3390/info16020087 · Cited by 80 · see citing works
Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments
Balse, R., Valaboju, B., Singhal, S., Warriem, J. M., & Prasad, P. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588852 · Cited by 63 · see citing works
Kuhail, M. A., Mathew, S. S., Khalil, A., Berengueres, J., & Shah, S. J. H. (2024) · Science of Computer Programming · doi:10.1016/j.scico.2024.103111 · Cited by 68 · see citing works
Can Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study
Shoufan, A. (2023) · ACM Transactions on Computing Education · doi:10.1145/3628162 · Cited by 59 · see citing works
Wang, T., Vargas-Díaz, D., Brown, C., & Chen, Y. (2023) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00018 · Cited by 68 · see citing works
Copiloting the future: How generative AI transforms Software Engineering
Banh, L., Holldack, F., & Strobel, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107751 · Cited by 66 · see citing works
Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study
Sheard, J., Denny, P., Hellas, A., Leinonen, J., Malmi, L., & Simon (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630880 · Cited by 54 · see citing works
Evaluating Contextually Personalized Programming Exercises Created with Generative AI
Logacheva, E., Hellas, A., Prather, J. E., Sarsa, S., & Leinonen, J. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671103 · Cited by 63 · see citing works
Budhiraja, R., Joshi, I., Challa, J. S., Akolekar, H. D., & Kumar, D. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636257 · Cited by 52 · see citing works
Chen, J., Lu, X., Du, Y., Rejtig, M., Bagley, R., Horn, M., & Wilensky, U. J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642377 · Cited by 55 · see citing works
GAIDE: A Framework for Using Generative AI to Assist in Course Content Development
Dickey, E., & Bejarano, A. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893132 · Cited by 49 · see citing works
Generative AI in the context of assistive technologies: Trends, limitations and future directions
Fu, B., Hadid, A., & Damer, N. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105347 · Cited by 52 · see citing works
Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation
Godoy, W. F., Valero‐Lara, P., Teranishi, K., Balaprakash, P., & Vetter, J. S. (2023) · International Conference on Parallel Processing Workshops (ICPPW) · doi:10.1145/3605731.3605886 · Cited by 42 · see citing works
Software engineering education in the era of conversational AI: current trends and future directions
Sengul, C., Neykova, R., & Destefanis, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1436350 · Cited by 55 · see citing works
Liu, J., & Li, S. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241240460 · Cited by 40 · see citing works
Chugh, R., Turnbull, D., Morshed, A., Sabrina, F., Azad, S., Rashid, M. M., Kaisar, S., & Subramani, S. (2025) · Computer Applications in Engineering Education · doi:10.1002/cae.70002 · Cited by 50 · see citing works
Ma, Y., Zheng, S., Yang, Z., Zheng, P., Leng, J., & Hong, J. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.07.019 · Cited by 41 · see citing works
Weisz, J. D., Kumar, S. V., Muller, M., Browne, K.-E., Goldberg, A., Heintze, K. E., & Bajpai, S. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706670 · Cited by 37 · see citing works
Evaluating the Quality of LLM-Generated Explanations for Logical Errors in CS1 Student Programs
Balse, R., Kumar, V., Prasad, P., & Warriem, J. M. (2023) · Bangalore Annual Compute Conference · doi:10.1145/3627217.3627233 · Cited by 34 · see citing works
Abolnejadian, M., Alipour, S., & Taeb, K. (2024) · CHI Extended Abstracts · doi:10.1145/3613905.3637148 · Cited by 39 · see citing works
Dickey, E., Bejarano, A., & Garg, C. (2024) · SN Computer Science · doi:10.1007/s42979-024-03074-y · Cited by 33 · see citing works
Lohr, D., Keuning, H., & Kiesler, N. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13107 · Cited by 33 · see citing works
Humble, N. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00494-x · Cited by 40 · see citing works
Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models
MacNeil, S., Denny, P. C., Tran, A., Leinonen, J., Bernstein, S., Hellas, A., Sarsa, S., & Kim, J. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636245 · Cited by 40 · see citing works
Akolekar, H. D., Jhamnani, P., Kumar, V., Tailor, V., Pote, A., Meena, A., Kumar, K., Challa, J. S., & Kumar, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-93871-z · Cited by 43 · see citing works
Hou, I., Man, O., Mettille, S., Gutierrez, S., Angelikas, K., & MacNeil, S. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636247 · Cited by 31 · see citing works
Mahon, J., Namee, B. M., & Becker, B. A. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653602 · Cited by 30 · see citing works
Exploring Gen-AI applications in building research and industry: A review
Wan, H., Zhang, J., Chen, Y., Xu, W., & Feng, F. (2025) · Building Simulation · doi:10.1007/s12273-025-1279-x · Cited by 37 · see citing works
An Industry Case Study on Adoption of AI-based Programming Assistants
Davila, N., Wiese, I., Steinmacher, I., da Silva, L. L., Kawamoto, A. L. S., Favaro, G. J. P., & Nunes, I. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3639477.3643648 · Cited by 30 · see citing works
Literature Review on the Integration of Generative AI in Programming Education
Nathaniel, J., Oyelere, S. S., Suhonen, J. T., & Tedre, M. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00524-3 · Cited by 40 · see citing works
Always Provide Context: The Effects of Code Context on Programming Error Message Enhancement
Santos, E. A., Prasad, P., & Becker, B. A. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617909 · Cited by 27 · see citing works
Prather, J. E., Denny, P. C., Leinonen, J., Becker, B. A., Al-Bluwi, I., Caspersen, M. E., Craig, M., Keuning, H., Kiesler, N., Kohn, T., et al. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587103.3594206 · Cited by 26 · see citing works
Nathaniel, J., Oyelere, S. S., Suhonen, J. T., & Tedre, M. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100460 · Cited by 37 · see citing works
Atcheson, A., Khan, O., Siemann, B., Jain, A., & Karahalios, K. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714121 · Cited by 22 · see citing works
Maher, M. L., Tadimalla, S. Y., & Dhamani, D. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343037 · Cited by 22 · see citing works
Large language model evaluation for high‐performance computing software development
Godoy, W. F., Valero‐Lara, P., Teranishi, K., Balaprakash, P., & Vetter, J. S. (2024) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.8269 · Cited by 20 · see citing works
Generative Artificial Intelligence in Instructional System Design
Chai, D. S., Kim, H. S., Kim, K. N., Ha, Y., Shin, S. S. H., & Yoon, S. W. (2025) · Human Resource Development Review · doi:10.1177/15344843251320256 · Cited by 20 · see citing works
Students' Use of GitHub Copilot for Working with Large Code Bases
Shah, A., Chernova, A., Tomson, E., Porter, L., Griswold, W. G., & Soosai Raj, A. G. (2025) · Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3641554.3701800 · Cited by 19 · see citing works
Lauren, P., & Watta, P. (2023) · Frontiers in Education Conference · doi:10.1109/fie58773.2023.10342970 · Cited by 12 · see citing works
Rug: Turbo Llm for Rust Unit Test Generation
Cheng, X. C., Sang, F., Zhai, Y., Zhang, X., & Kim, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00097 · Cited by 17 · see citing works
An Experiment on Leveraging ChatGPT for Online Teaching and Assessment of Database Students
Carr, N., Shawon, F. R., & Jamil, H. M. (2023) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale56641.2023.10398239 · Cited by 16 · see citing works
An assessment of large language models for OpenMP-based code parallelization: a user perspective
Mišić, M. J., & Dodović, M. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-01019-z · Cited by 16 · see citing works
Insights from the Frontline: GenAI Utilization Among Software Engineering Students
Choudhuri, R., Ramakrishnan, A., Chatterjee, A., Trinkenreich, B., Steinmacher, I., Gerosa, M. A., & Sarma, A. (2025) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet66350.2025.00007 · Cited by 20 · see citing works
Human-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 26 · see citing works
Analyzing LLM Usage in an Advanced Computing Class in India
Arora, U., Garg, A., Gupta, A., Jain, S., Mehta, R. L., Oberoi, R., Prachi, Raina, A., Saini, M., Sharma, S., et al. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716657 · Cited by 15 · see citing works
Analysis of Generative AI Policies in Computing Course Syllabi
Ali, A., Collier, A. H., Dewan, U., McDonald, N., & Johri, A. (2025) · Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3641554.3701823 · Cited by 14 · see citing works
Don’t settle for the first! How many GitHub Copilot solutions should you check?
Oertel, J., Klünder, J., & Hebig, R. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107737 · Cited by 14 · see citing works
Acceptance Test Generation with Large Language Models: An Industrial Case Study
Ferreira, M., Viegas, L. S., Faria, J. P., & Lima, B. (2025) · doi:10.1109/ast66626.2025.00007 · Cited by 18 · see citing works
Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions
Lehtinen, T., Koutcheme, C., & Hellas, A. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640058 · Cited by 14 · see citing works
Smith, D. H., & Zilles, C. B. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653582 · Cited by 14 · see citing works
Bull, C., & Kharrufa, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.13936 · Cited by 14 · see citing works
Cipriano, B. P., & Alves, P. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640052 · Cited by 13 · see citing works
Mailach, A., Gorgosch, D., Siegmund, N., & Siegmund, J. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10561-6 · Cited by 13 · see citing works
Hussein, R., Zhang, Z., Amarante, P., Hancock, N., Orduña, P., & Rodríguez-Gil, L. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892918 · Cited by 13 · see citing works
Assessing and Analyzing the Correctness of GitHub Copilot's Code Suggestions
Mo, R., Wang, D., Zhan, W., Jiang, Y., Wang, Y., Zhao, Y., Li, Z., & Ma, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715108 · Cited by 11 · see citing works
Kohn, T. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70043 · Cited by 13 · see citing works
Large Language Models (LLMs) in Programming Learning: The Current Research State and Agenda
Fu, Q., Zhao, Y., Jia, Z., & Zheng, Y. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3622043 · Cited by 10 · see citing works
Teaching IT Software Fundamentals: Strategies and Techniques for Inclusion of Large Language Models
Gumina, S., Dalton, T., & Gerdes, J. H. (2023) · Conference on Information Technology Education · doi:10.1145/3585059.3611409 · Cited by 9 · see citing works
A Bug's New Life: Creating Refute Questions from Filtered CS1 Student Code Snapshots
Agarwal, N., Kumar, V., Raman, A., & Karkare, A. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617916 · Cited by 9 · see citing works
The Robots are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00658 · Cited by 9 · see citing works
Assessment of ChatGPT's Proficiency in Software Development
Kim, D., Chen, J., Ming, H., & Lu, L. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00421 · Cited by 8 · see citing works
Li, N., Broner, S., Kim, Y., Mizuo, K., Sauder, E., To, C., Wang, A., Gila, O., & Shindler, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701946 · Cited by 8 · see citing works
Kim, D., & Ming, H. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107787 · Cited by 8 · see citing works
Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models
Zhang, K., Wang, S., Han, J., Zhu, X., Li, X., Wang, S., & Wen, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00041 · Cited by 10 · see citing works
Using Generative AI to Support UX Design Students in Web Development Courses
Buendía, F., & Piris-Ruano, J. (2025) · Applied Sciences · doi:10.3390/app15137389 · Cited by 8 · see citing works
Can Developers Prompt? A Controlled Experiment for Code Documentation Generation
Kruse, H.-A., Puhlfürß, T., & Maalej, W. (2024) · 2024 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme58944.2024.00058 · Cited by 8 · see citing works
Probeable Problems for Beginner-level Programming-with-AI Contests
Pawagi, M., & Kumar, V. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671108 · Cited by 7 · see citing works
Kumar, S. S., Lones, M. A., Maarek, M., & Zantout, H. (2024) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3643795.3648380 · Cited by 7 · see citing works
Viveros-Muñoz, R., Carrasco-Sáez, J. L., Contreras-Saavedra, C., San-Martín-Quiroga, S., & Contreras-Saavedra, C. E. (2025) · Applied Sciences · doi:10.3390/app15073882 · Cited by 10 · see citing works
Gutierrez, S., Hou, I., Lee, J., Angelikas, K., Man, O., Mettille, S., Prather, J. E., Denny, P. C., & MacNeil, S. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716654 · Cited by 7 · see citing works
A Systematic Review of Gen-AI Applications in Education: Rewards, Challenges and Future Prospects
Wanyonyi, E. N., & Murithi, M. (2025) · Pan-African journal of education and social sciences · doi:10.56893/pajes2025v06i01.01 · Cited by 7 · see citing works
Fostering Responsible AI Use Through Negative Expertise: A Contextualized Autocompletion Quiz
MacNeil, S., Prather, J. E., Nabid, R. A., Gutierrez, S., Carvalho, S., Shrestha, S., Denny, P. C., Reeves, B. N., Leinonen, J., & Rossetti, R. L. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729067 · Cited by 7 · see citing works
Stone, I. (2023) · Conference on Human Centred Artificial Intelligence - Education and Practice (HCAIep) · doi:10.1145/3633083.3633099 · Cited by 7 · see citing works
Evaluating Copilot on CS1 Code Writing Problems with Suppressed Specifications
Venkatesh, V., Venkatesh, V., & Kumar, V. (2023) · Bangalore Annual Compute Conference · doi:10.1145/3627217.3627235 · Cited by 7 · see citing works
Trust and Reliance in Evolving Human-AI Workflows (TREW)
Ashktorab, Z., Bansal, G., Buçinca, Z., Holstein, K., Hullman, J. R., Smith, A., Wu, T., & Zhang, W. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3636319 · Cited by 8 · see citing works
Exploring AI Integration in Software Development: Case Studies and Insights
Udoidiok, I., Reza, H., & Zhang, J. (2024) · IEEE National Aerospace and Electronics Conference (NAECON) · doi:10.1109/naecon61878.2024.10670347 · Cited by 7 · see citing works
Yılmaz, F. G. K., & Yılmaz, R. (2024) · Journal of Computing in Higher Education · doi:10.1007/s12528-024-09422-4 · Cited by 7 · see citing works
Begemann, A., & Hutson, J. (2025) · Journal of Information Economics · doi:10.58567/jie03010001 · Cited by 7 · see citing works
Cognitive Apprenticeship and Artificial Intelligence Coding Assistants
Poitras, E., Crane, B. G. C., Dempsey, D., Bragg, T. A., Siegel, A. A., & Lin, M. P.-C. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-1066-3.ch013 · Cited by 7 · see citing works
Šavelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09325 · Cited by 7 · see citing works
Generative AI’s Aid in Business Process Modeling Instructional Design: A Case Study
Maslov, I., Poelmans, S., Wautelet, Y., & Rosenthal, K. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-032-02936-2_24 · Cited by 6 · see citing works
Engineering Students' Experiences With ChatGPT to Generate Code for Disciplinary Programming
Vieira, C., Hoz, J. L. D. L., Magana, A. J., & Restrepo, D. (2025) · Computer Applications in Engineering Education · doi:10.1002/cae.70090 · Cited by 10 · see citing works
Uma análise do uso de ferramentas de geração de código por alunos de computação
Lira, W. A. L., Neto, P. D. A. D. S., & Osório, L. F. M. (2024) · doi:10.5753/educomp.2024.237427 · Cited by 6 · see citing works
Aguiar, L., Paixão, M. H. E., do Carmo, R. A. F., Soares, E. L. C., Leal, A., Freitas, M., & Gama, E. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3690755 · Cited by 6 · see citing works
Compiler-Integrated, Conversational AI for Debugging CS1 Programs
Renzella, J., Vassar, A., Solano, L. L., & Taylor, A. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701827 · Cited by 5 · see citing works
The Recent Trends of Research on GitHub Copilot: A Systematic Review
Ani, Z. C., Hamid, Z. A., & Zhamri, N. N. (2024) · Communications in Computer and Information Science · doi:10.1007/978-981-99-9589-9_27 · Cited by 5 · see citing works
Comparing Proficiency of ChatGPT and Bard in Software Development
Kim, D. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_2 · Cited by 4 · see citing works
- Authors
- Wermelinger, M.
- Affiliations
- The Open University, Milton Keynes, United Kingdom
- ORCID
- Wermelinger, Michel
- Venue
- Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2023-03-02
- Pages
- 172-178
- OpenAlex
- W4324138978
- Cited by
- 191
- References
- 10
- Citations by year
- 2026: 35 · 2025: 65 · 2024: 51 · 2023: 40
- FWCI
- 57.69
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- Open Research Online (The Open University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- GitHub Copilot, OpenAI Codex, large language models, programming education, introductory programming, academic integrity, code generation, code explanation, test case generation, bug fixes, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
