The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Using GitHub Copilot to Solve Simple Programming Problems
Open accessAbstract
The teaching and assessment of introductory programming involves writing code that solves a problem described by text. Previous research found that OpenAI's Codex, a natural language machine learning model trained on billions of lines of code, performs well on many programming problems, often generating correct and readable Python code. GitHub's version of Codex, Copilot, is freely available to students. This raises pedagogic and academic integrity concerns. Educators need to know what Copilot is capable of, in order to adapt their teaching to AI-powered programming assistants. Previous research evaluated the most performant Codex model quantitatively, e.g. how many problems have at least one correct suggestion that passes all tests. Here I evaluate Copilot instead, to see if and how it differs from Codex, and look qualitatively at the generated suggestions, to understand the limitations of Copilot. I also report on the experience of using Copilot for other activities asked of students in programming courses: explaining code, generating tests and fixing bugs. The paper concludes with a discussion of the implications of the observed capabilities for the teaching of programming.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given the ability of GenAI tools to generate human-like content, such as language and code, concerns regarding academic honesty and student learning have dominated discourse [16, 29, 33, 36, 55, 115, 117].”
From "I'd Never Actually Realized How Big An Impact It Had Until Now": Perspectives of University Students with Disabilities on Generative Artificial Intelligence (2025) · cites this paper as [115]
Experimental — accuracy not yet measuredIn the citing paper
“Wermelinger [191] speculated that students may spend excessive time on prompt engineering in the hope of hitting on a successful solution rather than making progress towards the solution, which was indeed observed in the study by Prather et al. [159] in the 'shepherding' interaction pattern.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
In the citing paper
“Automatically generated tests may be partially complete, leading students to inadequately test their code [191].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
In the citing paper
“Wermelinger [191] observed that explanations generated by Copilot focused on a line-by-line description of what the code did rather than how it achieved the goal desired.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
In the citing paper
“The solutions generated by AI might not be easily identified as AI-generated [160], which requires teachers to adapt and develop new teaching strategies to ensure academic integrity is maintained [191].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
In the citing paper
“Although LLM-based tools like Copilot may serve as a useful springboard for solving CS1 problems, students still need to dedicate time to learning algorithmic thinking, program comprehension, debugging, and communication skills [191] in order to become not only proficient computing experts but also to use LLMs effectively.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
In the citing paper
“This might be due to the use of a relatively high temperature value for these models-previous work found similar results and speculated that Copilot likely uses a lower temperature value [191].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Learning to program = learning to construct mechanisms and explanations
Soloway, E. (1986) · CACM · doi:10.1145/6592.6594 · Cited by 615
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
Novice programmer errors: language constructs and plan composition
Ebrahimi, A. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1069 · Cited by 155
Lobb, R., & Harlow, J. (2016) · ACM Inroads · doi:10.1145/2810041 · Cited by 142
Pattern-oriented instruction and its influence on problem decomposition and solution construction
Muller, O., Ginat, D., & Haberman, B. (2007) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1268784.1268830 · Cited by 113
The recurring rainfall problem
Fisler, K. (2014) · doi:10.1145/2632320.2632346 · Cited by 112
Soloway's Rainfall Problem Has Become Harder
Simon (2013) · International Conference on Learning and Teaching in Computing and Engineering (LaTICE) · doi:10.1109/latice.2013.44 · Cited by 61
Research works printed but not linked: 2
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. https://doi.org/10.48550/ARXIV.2107.03374 · DOI · Paper page
- Yujia Li David Choi Junyoung Chung Nate Kushman Julian Schrittwieser Rémi Leblond Tom Eccles James Keeling Felix Gimeno Agustin Dal Lago Thomas Hubert Peter Choy Cyprien de Masson d'Autume Igor Babuschkin Xinyun Chen Po-Sen Huang Johannes Welbl Sven Gowal Alexey Cherepanov James Molloy Daniel J. Mankowitz Esme Sutherland Robson Pushmeet Kohli Nando de Freitas Koray Kavukcuoglu and Oriol Vinyals. 2022. Competition-Level Code Generation with AlphaCode. https://doi.org/10.48550/ARXIV.2203.07814 · DOI · Paper page
Cited by
Sorted by citations
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“Wermelinger [106] evaluated the capabilities of Copilot in the spirit of Finnie-Ainsley et al. [42] who evaluated Codex and found it to still miss the mark more often than not.”
MethodLau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241
Navigating the Complexity of Generative AI Adoption in Software Engineering
Russo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3652154 · Cited by 217
“While Wermelinger explored the implications of Copilot on teaching and assessment methods in programming courses [97], Dakhel et al.”
BackgroundExploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145
The Impact of Large Language Models on Programming Education and Student Learning Outcomes
Jošt, G., Taneski, V., & Karakatič, S. (2024) · Applied Sciences · doi:10.3390/app14104115 · Cited by 143
Generative AI in Computing Education: Perspectives of Students and Instructors
Zastudil, C., Rogalska, M., Kapp, C., Vaughn, J., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343467 · Cited by 136
Savelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588792 · Cited by 109
Bull, C., & Kharrufa, A. (2024) · IEEE Software · doi:10.1109/ms.2023.3300574 · Cited by 82
Building machines that learn and think with people
Collins, K. M., Sucholutsky, I., Bhatt, U., Chandra, K., Wong, L., Lee, M., Zhang, C. E., Zhi‐Xuan, T., Ho, M. K., Mansinghka, V. K., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01991-9 · Cited by 125
ChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course
Popovici, M.-D. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2269006 · Cited by 70
Kuhail, M. A., Mathew, S. S., Khalil, A., Berengueres, J., & Shah, S. J. H. (2024) · Science of Computer Programming · doi:10.1016/j.scico.2024.103111 · Cited by 68
Transitioning from MLOps to LLMOps: Navigating the Unique Challenges of Large Language Models
Pahune, S. A., & Akhtar, Z. (2025) · Information · doi:10.3390/info16020087 · Cited by 80
Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments
Balse, R., Valaboju, B., Singhal, S., Warriem, J. M., & Prasad, P. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588852 · Cited by 63
Can Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study
Shoufan, A. (2023) · ACM Transactions on Computing Education · doi:10.1145/3628162 · Cited by 59
Wang, T., Vargas-Díaz, D., Brown, C., & Chen, Y. (2023) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00018 · Cited by 80
Copiloting the future: How generative AI transforms Software Engineering
Banh, L., Holldack, F., & Strobel, G. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107751 · Cited by 78
Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study
Sheard, J., Denny, P., Hellas, A., Leinonen, J., Malmi, L., & Simon (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630880 · Cited by 54
Evaluating Contextually Personalized Programming Exercises Created with Generative AI
Logacheva, E., Hellas, A., Prather, J. E., Sarsa, S., & Leinonen, J. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671103 · Cited by 85
Budhiraja, R., Joshi, I., Challa, J. S., Akolekar, H. D., & Kumar, D. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636257 · Cited by 52
Chen, J., Lu, X., Du, Y., Rejtig, M., Bagley, R., Horn, M., & Wilensky, U. J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642377 · Cited by 66
GAIDE: A Framework for Using Generative AI to Assist in Course Content Development
Dickey, E., & Bejarano, A. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893132 · Cited by 92
Generative AI in the context of assistive technologies: Trends, limitations and future directions
Fu, B., Hadid, A., & Damer, N. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105347 · Cited by 57
Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation
Godoy, W. F., Valero‐Lara, P., Teranishi, K., Balaprakash, P., & Vetter, J. S. (2023) · International Conference on Parallel Processing Workshops (ICPPW) · doi:10.1145/3605731.3605886 · Cited by 52
Software engineering education in the era of conversational AI: current trends and future directions
Sengul, C., Neykova, R., & Destefanis, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1436350 · Cited by 63
Liu, J., & Li, S. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241240460 · Cited by 53
Chugh, R., Turnbull, D., Morshed, A., Sabrina, F., Azad, S., Rashid, M. M., Kaisar, S., & Subramani, S. (2025) · Computer Applications in Engineering Education · doi:10.1002/cae.70002 · Cited by 52
Ma, Y., Zheng, S., Yang, Z., Zheng, P., Leng, J., & Hong, J. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.07.019 · Cited by 48
Weisz, J. D., Kumar, S. V., Muller, M., Browne, K.-E., Goldberg, A., Heintze, K. E., & Bajpai, S. (2025) · Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706599.3706670 · Cited by 37
Evaluating the Quality of LLM-Generated Explanations for Logical Errors in CS1 Student Programs
Balse, R., Kumar, V., Prasad, P., & Warriem, J. M. (2023) · Bangalore Annual Compute Conference · doi:10.1145/3627217.3627233 · Cited by 42
Abolnejadian, M., Alipour, S., & Taeb, K. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3637148 · Cited by 49
Dickey, E., Bejarano, A., & Garg, C. (2024) · SN Computer Science · doi:10.1007/s42979-024-03074-y · Cited by 33
Lohr, D., Keuning, H., & Kiesler, N. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13107 · Cited by 51
Humble, N. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00494-x · Cited by 48
Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models
MacNeil, S., Denny, P. C., Tran, A., Leinonen, J., Bernstein, S., Hellas, A., Sarsa, S., & Kim, J. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636245 · Cited by 49
Akolekar, H. D., Jhamnani, P., Kumar, V., Tailor, V., Pote, A., Meena, A., Kumar, K., Challa, J. S., & Kumar, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-93871-z · Cited by 49
Hou, I., Man, O., Mettille, S., Gutierrez, S., Angelikas, K., & MacNeil, S. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636247 · Cited by 46
Mahon, J., Namee, B. M., & Becker, B. A. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653602 · Cited by 47
Exploring Gen-AI applications in building research and industry: A review
Wan, H., Zhang, J., Chen, Y., Xu, W., & Feng, F. (2025) · Building Simulation · doi:10.1007/s12273-025-1279-x · Cited by 42
An Industry Case Study on Adoption of AI-based Programming Assistants
Davila, N., Wiese, I., Steinmacher, I., da Silva, L. L., Kawamoto, A. L. S., Favaro, G. J. P., & Nunes, I. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3639477.3643648 · Cited by 48
Literature Review on the Integration of Generative AI in Programming Education
Nathaniel, J., Oyelere, S. S., Suhonen, J. T., & Tedre, M. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00524-3 · Cited by 45
Always Provide Context: The Effects of Code Context on Programming Error Message Enhancement
Santos, E. A., Prasad, P., & Becker, B. A. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617909 · Cited by 34
Prather, J. E., Denny, P. C., Leinonen, J., Becker, B. A., Al-Bluwi, I., Caspersen, M. E., Craig, M., Keuning, H., Kiesler, N., Kohn, T., et al. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587103.3594206 · Cited by 26
Nathaniel, J., Oyelere, S. S., Suhonen, J. T., & Tedre, M. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100460 · Cited by 42
Atcheson, A., Khan, O., Siemann, B., Jain, A., & Karahalios, K. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714121 · Cited by 22
Maher, M. L., Tadimalla, S. Y., & Dhamani, D. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343037 · Cited by 33
Large language model evaluation for high‐performance computing software development
Godoy, W. F., Valero‐Lara, P., Teranishi, K., Balaprakash, P., & Vetter, J. S. (2024) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.8269 · Cited by 29
Generative Artificial Intelligence in Instructional System Design
Chai, D. S., Kim, H. S., Kim, K. N., Ha, Y., Shin, S. S. H., & Yoon, S. W. (2025) · Human Resource Development Review · doi:10.1177/15344843251320256 · Cited by 20
Students' Use of GitHub Copilot for Working with Large Code Bases
Shah, A., Chernova, A., Tomson, E., Porter, L., Griswold, W. G., & Soosai Raj, A. G. (2025) · Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3641554.3701800 · Cited by 19
Lauren, P., & Watta, P. B. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342970 · Cited by 27
Rug: Turbo Llm for Rust Unit Test Generation
Cheng, X. C., Sang, F., Zhai, Y., Zhang, X., & Kim, T. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00097 · Cited by 26
An Experiment on Leveraging ChatGPT for Online Teaching and Assessment of Database Students
Carr, N., Shawon, F. R., & Jamil, H. M. (2023) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale56641.2023.10398239 · Cited by 16
An assessment of large language models for OpenMP-based code parallelization: a user perspective
Mišić, M. J., & Dodović, M. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-01019-z · Cited by 18
Insights from the Frontline: GenAI Utilization Among Software Engineering Students
Choudhuri, R., Ramakrishnan, A., Chatterjee, A., Trinkenreich, B., Steinmacher, I., Gerosa, M. A., & Sarma, A. (2025) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet66350.2025.00007 · Cited by 23
Human-AI experience in integrated development environments: a systematic literature review
Sergeyuk, A., Zakharov, I., Koshchenko, E., & Izadi, M. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10793-0 · Cited by 38
Analyzing LLM Usage in an Advanced Computing Class in India
Arora, U., Garg, A., Gupta, A., Jain, S., Mehta, R. L., Oberoi, R., Prachi, Raina, A., Saini, M., Sharma, S., et al. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716657 · Cited by 43
Analysis of Generative AI Policies in Computing Course Syllabi
Ali, A., Collier, A. H., Dewan, U., McDonald, N., & Johri, A. (2025) · Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3641554.3701823 · Cited by 14
Don’t settle for the first! How many GitHub Copilot solutions should you check?
Oertel, J., Klünder, J., & Hebig, R. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107737 · Cited by 17
Acceptance Test Generation with Large Language Models: An Industrial Case Study
Ferreira, M., Viegas, L. S., Faria, J. P., & Lima, B. (2025) · doi:10.1109/ast66626.2025.00007 · Cited by 18
Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions
Lehtinen, T., Koutcheme, C., & Hellas, A. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640058 · Cited by 24
Smith, D. H., & Zilles, C. B. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653582 · Cited by 25
Bull, C., & Kharrufa, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.13936 · Cited by 14
Cipriano, B. P., & Alves, P. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640052 · Cited by 21
Mailach, A., Gorgosch, D., Siegmund, N., & Siegmund, J. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10561-6 · Cited by 13
Hussein, R., Zhang, Z., Amarante, P., Hancock, N., Orduña, P., & Rodríguez-Gil, L. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892918 · Cited by 13
Assessing and Analyzing the Correctness of GitHub Copilot's Code Suggestions
Mo, R., Wang, D., Zhan, W., Jiang, Y., Wang, Y., Zhao, Y., Li, Z., & Ma, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715108 · Cited by 11
Kohn, T. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70043 · Cited by 14
Large Language Models (LLMs) in Programming Learning: The Current Research State and Agenda
Fu, Q., Zhao, Y., Jia, Z., & Zheng, Y. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3622043 · Cited by 11
Teaching IT Software Fundamentals: Strategies and Techniques for Inclusion of Large Language Models
Gumina, S., Dalton, T., & Gerdes, J. H. (2023) · Conference on Information Technology Education · doi:10.1145/3585059.3611409 · Cited by 9
A Bug's New Life: Creating Refute Questions from Filtered CS1 Student Code Snapshots
Agarwal, N., Kumar, V., Raman, A., & Karkare, A. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617916 · Cited by 9
The Robots are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00658 · Cited by 9
Assessment of ChatGPT's Proficiency in Software Development
Kim, D., Chen, J., Ming, H., & Lu, L. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00421 · Cited by 8
Li, N., Broner, S., Kim, Y., Mizuo, K., Sauder, E., To, C., Wang, A., Gila, O., & Shindler, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701946 · Cited by 12
Kim, D., & Ming, H. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107787 · Cited by 10
Your Fix Is My Exploit: Enabling Comprehensive DL Library API Fuzzing with Large Language Models
Zhang, K., Wang, S., Han, J., Zhu, X., Li, X., Wang, S., & Wen, S. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00041 · Cited by 17
Using Generative AI to Support UX Design Students in Web Development Courses
Buendía, F., & Piris-Ruano, J. (2025) · Applied Sciences · doi:10.3390/app15137389 · Cited by 8
Can Developers Prompt? A Controlled Experiment for Code Documentation Generation
Kruse, H.-A., Puhlfürß, T., & Maalej, W. (2024) · 2024 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme58944.2024.00058 · Cited by 8
Probeable Problems for Beginner-level Programming-with-AI Contests
Pawagi, M., & Kumar, V. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671108 · Cited by 10
Kumar, S. S., Lones, M. A., Maarek, M., & Zantout, H. (2024) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3643795.3648380 · Cited by 7
Viveros-Muñoz, R., Carrasco-Sáez, J. L., Contreras-Saavedra, C., San-Martín-Quiroga, S., & Contreras-Saavedra, C. E. (2025) · Applied Sciences · doi:10.3390/app15073882 · Cited by 10
Gutierrez, S., Hou, I., Lee, J., Angelikas, K., Man, O., Mettille, S., Prather, J. E., Denny, P. C., & MacNeil, S. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716654 · Cited by 15
A Systematic Review of Gen-AI Applications in Education: Rewards, Challenges and Future Prospects
Wanyonyi, E. N., & Murithi, M. (2025) · Pan-African journal of education and social sciences · doi:10.56893/pajes2025v06i01.01 · Cited by 7
Fostering Responsible AI Use Through Negative Expertise: A Contextualized Autocompletion Quiz
MacNeil, S., Prather, J. E., Nabid, R. A., Gutierrez, S., Carvalho, S., Shrestha, S., Denny, P. C., Reeves, B. N., Leinonen, J., & Rossetti, R. L. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729067 · Cited by 14
Stone, I. (2023) · Conference on Human Centred Artificial Intelligence - Education and Practice (HCAIep) · doi:10.1145/3633083.3633099 · Cited by 9
Evaluating Copilot on CS1 Code Writing Problems with Suppressed Specifications
Venkatesh, V., Venkatesh, V., & Kumar, V. (2023) · Bangalore Annual Compute Conference · doi:10.1145/3627217.3627235 · Cited by 7
Trust and Reliance in Evolving Human-AI Workflows (TREW)
Ashktorab, Z., Bansal, G., Buçinca, Z., Holstein, K., Hullman, J. R., Smith, A., Wu, T., & Zhang, W. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3636319 · Cited by 9
Exploring AI Integration in Software Development: Case Studies and Insights
Udoidiok, I., Reza, H., & Zhang, J. (2024) · IEEE National Aerospace and Electronics Conference (NAECON) · doi:10.1109/naecon61878.2024.10670347 · Cited by 10
Yılmaz, F. G. K., & Yılmaz, R. (2024) · Journal of Computing in Higher Education · doi:10.1007/s12528-024-09422-4 · Cited by 7
Begemann, A., & Hutson, J. (2025) · Journal of Information Economics · doi:10.58567/jie03010001 · Cited by 11
Cognitive Apprenticeship and Artificial Intelligence Coding Assistants
Poitras, E., Crane, B. G. C., Dempsey, D., Bragg, T. A., Siegel, A. A., & Lin, M. P.-C. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-1066-3.ch013 · Cited by 7
Šavelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09325 · Cited by 7
Generative AI’s Aid in Business Process Modeling Instructional Design: A Case Study
Maslov, I., Poelmans, S., Wautelet, Y., & Rosenthal, K. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-032-02936-2_24 · Cited by 6
Engineering Students' Experiences With ChatGPT to Generate Code for Disciplinary Programming
Vieira, C., Hoz, J. L. D. L., Magana, A. J., & Restrepo, D. (2025) · Computer Applications in Engineering Education · doi:10.1002/cae.70090 · Cited by 10
Uma análise do uso de ferramentas de geração de código por alunos de computação
Lira, W. A. L., Neto, P. D. A. D. S., & Osório, L. F. M. (2024) · doi:10.5753/educomp.2024.237427 · Cited by 8
Aguiar, L., Paixão, M. H. E., do Carmo, R. A. F., Soares, E. L. C., Leal, A., Freitas, M., & Gama, E. (2024) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/3674805.3690755 · Cited by 6
Compiler-Integrated, Conversational AI for Debugging CS1 Programs
Renzella, J., Vassar, A., Solano, L. L., & Taylor, A. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701827 · Cited by 15
The Recent Trends of Research on GitHub Copilot: A Systematic Review
Ani, Z. C., Hamid, Z. A., & Zhamri, N. N. (2024) · Communications in Computer and Information Science · doi:10.1007/978-981-99-9589-9_27 · Cited by 5
Comparing Proficiency of ChatGPT and Bard in Software Development
Kim, D. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_2 · Cited by 4
Metadata
- Authors
- Wermelinger, M.
- Affiliations
- The Open University, Milton Keynes, United Kingdom
- ORCID
- Wermelinger, Michel
- Venue
- Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2023-03-02
- Pages
- 172-178
- OpenAlex
- W4324138978
- Cited by
- 191
- References
- 10
- Citations by year
- 2026: 35 · 2025: 65 · 2024: 51 · 2023: 40
- FWCI
- 57.69
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- Open Research Online (The Open University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- GitHub Copilot, OpenAI Codex, large language models, programming education, introductory programming, academic integrity, code generation, code explanation, test case generation, bug fixes, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
