The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Using GitHub Copilot to Solve Simple Programming Problems
Open accessAbstract
The teaching and assessment of introductory programming involves writing code that solves a problem described by text. Previous research found that OpenAI's Codex, a natural language machine learning model trained on billions of lines of code, performs well on many programming problems, often generating correct and readable Python code. GitHub's version of Codex, Copilot, is freely available to students. This raises pedagogic and academic integrity concerns. Educators need to know what Copilot is capable of, in order to adapt their teaching to AI-powered programming assistants. Previous research evaluated the most performant Codex model quantitatively, e.g. how many problems have at least one correct suggestion that passes all tests. Here I evaluate Copilot instead, to see if and how it differs from Codex, and look qualitatively at the generated suggestions, to understand the limitations of Copilot. I also report on the experience of using Copilot for other activities asked of students in programming courses: explaining code, generating tests and fixing bugs. The paper concludes with a discussion of the implications of the observed capabilities for the teaching of programming.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given the ability of GenAI tools to generate human-like content, such as language and code, concerns regarding academic honesty and student learning have dominated discourse [16, 29, 33, 36, 55, 115, 117].”
From "I'd Never Actually Realized How Big An Impact It Had Until Now": Perspectives of University Students with Disabilities on Generative Artificial Intelligence (2025) · cites this paper as [115]
Experimental — accuracy not yet measuredIn the citing paper
“Wermelinger [191] speculated that students may spend excessive time on prompt engineering in the hope of hitting on a successful solution rather than making progress towards the solution, which was indeed observed in the study by Prather et al. [159] in the 'shepherding' interaction pattern.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
In the citing paper
“Automatically generated tests may be partially complete, leading students to inadequately test their code [191].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
In the citing paper
“Wermelinger [191] observed that explanations generated by Copilot focused on a line-by-line description of what the code did rather than how it achieved the goal desired.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
In the citing paper
“The solutions generated by AI might not be easily identified as AI-generated [160], which requires teachers to adapt and develop new teaching strategies to ensure academic integrity is maintained [191].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
In the citing paper
“Although LLM-based tools like Copilot may serve as a useful springboard for solving CS1 problems, students still need to dedicate time to learning algorithmic thinking, program comprehension, debugging, and communication skills [191] in order to become not only proficient computing experts but also to use LLMs effectively.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
In the citing paper
“This might be due to the use of a relatively high temperature value for these models-previous work found similar results and speculated that Copilot likely uses a lower temperature value [191].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [191]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Learning to program = learning to construct mechanisms and explanations
Soloway, E. (1986) · CACM · doi:10.1145/6592.6594 · Cited by 615
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
Novice programmer errors: language constructs and plan composition
Ebrahimi, A. (1994) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1994.1069 · Cited by 155
Lobb, R., & Harlow, J. (2016) · ACM Inroads · doi:10.1145/2810041 · Cited by 142
Pattern-oriented instruction and its influence on problem decomposition and solution construction
Muller, O., Ginat, D., & Haberman, B. (2007) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1268784.1268830 · Cited by 113
The recurring rainfall problem
Fisler, K. (2014) · doi:10.1145/2632320.2632346 · Cited by 112
Soloway's Rainfall Problem Has Become Harder
Simon (2013) · International Conference on Learning and Teaching in Computing and Engineering (LaTICE) · doi:10.1109/latice.2013.44 · Cited by 61
Research works printed but not linked: 2
- Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. https://doi.org/10.48550/ARXIV.2107.03374 · DOI · Paper page
- Yujia Li David Choi Junyoung Chung Nate Kushman Julian Schrittwieser Rémi Leblond Tom Eccles James Keeling Felix Gimeno Agustin Dal Lago Thomas Hubert Peter Choy Cyprien de Masson d'Autume Igor Babuschkin Xinyun Chen Po-Sen Huang Johannes Welbl Sven Gowal Alexey Cherepanov James Molloy Daniel J. Mankowitz Esme Sutherland Robson Pushmeet Kohli Nando de Freitas Koray Kavukcuoglu and Oriol Vinyals. 2022. Competition-Level Code Generation with AlphaCode. https://doi.org/10.48550/ARXIV.2203.07814 · DOI · Paper page
Cited by
Sorted by citations
Items 101–193 of 193
Liu, H., & Li, L. (2024) · Materials Today Communications · doi:10.1016/j.mtcomm.2024.109809 · Cited by 7
Implications and Considerations of AI-Generative Tools for Higher Education Practices
Em, S., Mok, S., Sam, R., Pen, D., & Serey, M. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-7873-1.ch011 · Cited by 4
Munisamy, M., Osman, S. Z. M., & Sanmugam, M. (2024) · International Journal of Modern Education · doi:10.35631/ijmoe.620027 · Cited by 4
Understanding and Enhancing CS Students’ Interaction Experience with AI Coding Assistant Tools
Long, X., Tan, X. L., Zhu, Y., Jiang, J., & Zhang, L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3785479 · Cited by 6
Chen, N., Qiu, L. K., Wang, A. Z., Wang, Z., & Yang, Y. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790726 · Cited by 14
Quille, K., Alattyanyi, C., Becker, B. A., Faherty, R., Gordon, D., Harte, M., Hensman, S., Hofmann, M., García, J. J., Kuznetsov, A., et al. (2024) · Revista de Educación a Distancia (RED) · doi:10.6018/red.603001 · Cited by 3
A Prompt-Based Approach for Software Development
Hamdi, M., & Kim, L. D. (2023) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci62032.2023.00267 · Cited by 3
An AI-driven Approach for Tailoring Java Programming Instructions
Thiyagarajan, G., & Vijayalakshmi, V. (2025) · International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT) · doi:10.1109/idciot64235.2025.10914687 · Cited by 4
Trustworthiness of Large Language Models for Code
Khati, D. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion66252.2025.00063 · Cited by 3
Filho, L. C. P., de Souza, T. D. P. C., & de Paula, L. B. (2025) · Revista Brasileira de Informática na Educação · doi:10.5753/rbie.2025.4477 · Cited by 3
Extracting REST API Endpoints from Microservices Using LLM Agents
Chaplia, O., & Klym, H. (2024) · International Conference on Dependable Systems, Services and Technologies (DESSERT) · doi:10.1109/dessert65323.2024.11122242 · Cited by 2
Park, E. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10730-1 · Cited by 3
The Impact of AI Tools on the Student Programming Experience: Interactions, Attitudes & Perceptions
Tadimalla, S. Y., Maher, M. L., Najjar, N. A., & Simon, C. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328137 · Cited by 2
Creating Thorough Tests for AI-Generated Code is Hard
Singhal, S., & Kumar, V. (2023) · Bangalore Annual Compute Conference · doi:10.1145/3627217.3627238 · Cited by 2
Digital Rubber Duck: Leveraging Large Language Models for Extreme Programming
Elvira, T., Procko, T., Couder, J. O., & Ochoa, O. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00051 · Cited by 5
Beyond Accuracy and Robustness Metrics for Large Language Models for Code
Rodríguez-Cárdenas, D. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3639478.3639792 · Cited by 2
The Emerging Artifacts of Centralized Open-Code
Choksi, M. Z., Mandel, I., Widder, D. G., & Shvartzshnaider, Y. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3659019 · Cited by 3
Tackling Students' Coding Assignments with LLMs
Dingle, A., & Kruliš, M. (2024) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3643795.3648389 · Cited by 8
Applications of Artificial Intelligence for Enhanced Bug Detection in Software Development
Al-Shameri, Y. N. H. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-6745-2.ch008 · Cited by 2
Overview of Neuro-Symbolic Integration Frameworks
Bhuyan, B. P., Ramdane-Chérif, A., Singh, T. P., & Tomar, R. (2024) · Studies in computational intelligence · doi:10.1007/978-981-97-8171-3_10 · Cited by 2
dos Santos, N. S., Santana, D. D. O., & Pereira, C. P. (2025) · doi:10.5753/educomp.2025.5385 · Cited by 2
Assessing Code Understanding in LLMs
Laneve, C., Spanò, A., Ressi, D., Rossi, S., & Michele Bugliesi, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-95497-9_13 · Cited by 2
Mehmood, S., Janjua, U. I., & Ahmed, A. (2023) · International Conference on Frontiers of Information Technology (FIT) · doi:10.1109/fit60620.2023.00013 · Cited by 2
POTENCIALIDADES E DESAFIOS DO GITHUB COPILOT COMO FERRAMENTA DA INTELIGÊNCIA ARTIFICIAL
de Sena, J. W. P., Barreto, A. S., Barbosa, J. C. N., & Alves, K. (2024) · P2P E INOVAÇÃO · doi:10.21728/p2p.2024v10n2e-7031 · Cited by 1
dos Santos, N. S., & Pereira, C. P. (2024) · doi:10.5753/urca.2024.245620 · Cited by 1
Koç, O., Yücedağ, İ., & Şentürk, Ü. (2025) · Düzce Üniversitesi Bilim ve Teknoloji Dergisi · doi:10.29130/dubited.1554356 · Cited by 1
Pachón, C. G., Pedraza, C. A., & Ballesteros, D. M. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9080200 · Cited by 2
From Coders to Critics: Empowering Students Through Peer Assessment in the Age of AI Copilots
Berrezueta-Guzmán, J., Krusche, S., & Wagner, S. (2025) · International Symposium on Educational Technology · doi:10.1109/iset65607.2025.00022 · Cited by 4
Online Platform to Inform GenAI Misuse in Programming Through Code Differences
Karnalim, O. (2025) · International Journal of Distance Education Technologies · doi:10.4018/ijdet.397045 · Cited by 2
Transformation of QSAR by AI and ML
Gomatam, A., Khan, A. A., Coutinho, K., Maheshwari, S., Adani, V., Martis, E. A. F., & Coutinho, E. C. (2025) · Springer handbooks · doi:10.1007/978-3-031-81728-1_25 · Cited by 1
Karnalim, O., Santiadi, S., Natalia, D., Wijaya, J. S., Rupilu, G. E., Ivanayus, Sarean, A. R., & Hassabian, G. A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-07316-7_17 · Cited by 1
Can We Trust the AI Pair Programmer? Copilot for API Misuse Detection and Correction
Mondal, S., Roy, C. K., Wang, H., Arguello, J., & Mathan, S. (2025) · IEEE International Conference on Collaborative Advances in Software and Computing (CASCON) · doi:10.1109/cascon66301.2025.00025 · Cited by 1
Are They All Good? Evaluating the Quality of CoTs in LLM-Based Code Generation
Zhang, B., Zhang, L., Luo, Z., Du, Y., Liu, F., Wang, S., & Shi, L. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3676295 · Cited by 10
Zhang, H., Yang, Y., Zhang, Y., Qiu, B., & Yang, J. (2026) · BMC Psychology · doi:10.1186/s40359-026-05011-5 · Cited by 3
GuardRails: Automated Suggestions for Clarifying Ambiguous Purpose Statements
Pawagi, M., & Kumar, V. (2023) · Bangalore Annual Compute Conference · doi:10.1145/3627217.3627234 · Cited by 1
Miranda, P. C., & de Andrade, M. H. S. (2024) · doi:10.5753/cbsoft_estendido.2024.4100
Hoz, J. L. D. L., Restrepo, D., & Vieira, C. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893514 · Cited by 1
Hyseni, L. N., & Dermaku, A. (2025) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.1846 · Cited by 2
Geração de Dicas de Próximo Passo utilizando Large Language Models (LLMs)
Araújo, L., Araújo, K. F., Reis, E., da Silva, M. O., & Pinheiro, A. (2025) · doi:10.5753/educomp_estendido.2025.6594 · Cited by 1
Černý, M. (2025) · European Education · doi:10.1080/10564934.2025.2569466 · Cited by 1
Zhang, H., Yu, D., Zhang, L., Rong, G., Yu, Y., Shen, H., Zhang, H., Shao, D., & Kuang, H. (2025) · Asia-Pacific Symposium on Internetware · doi:10.1145/3755881.3755889 · Cited by 3
How do Copilot Suggestions Impact Developers' Frustration and Productivity?
Guglielmi, E., Arnoudova, V., Bavota, G., Oliveto, R., & Scalabrino, S. (2025) · arXiv · doi:10.48550/arxiv.2504.06808 · Cited by 1
“As such, paradoxically, they might disrupt developers’ cognitive flow, increase their frustration [7]–[9] (especially if the they are irrelevant or require significant adaptation), and reduce their productivity [10], [11].”
Checking Conformance to a Subset of the Python Language
Michel Wermelinger, B. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587103.3594155
Procko, T., & Collins, S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4571367
A review of AI-assisted motion control
Li, Y., Chen, H., & Tsung, F. (2023) · doi:10.1117/12.3013434
A Study of Coding Learning Amongst Children: Motivation and Learning Performance
Hanafi, H. F., Mustafa, W. A., Idris, M. N., Ghani, M. M., Alkhayyat, A. H., Lah, N. H. C., & Seng, W. Y. (2023) · International Conference on Engineering Technology and its Applications (IICETA) · doi:10.1109/iiceta57613.2023.10351439
Results of Guided Expert Interviews
Kiesler, N. (2023) · Springer eBooks · doi:10.1007/978-3-031-47148-3_9
Спажев, О. М., & Клочко, О. (2025) · Educational Dimension · doi:10.55056/ed.801
Längrich, M., Schulze, J., Dobler, H., & Haslwanter, J. D. H. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892951
Transformation in Human–Computer Interaction: The AI-Enabled NLP
Sharma, H., Anubha, A., & Narang, D. (2025) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-83549-852-120251003 · Cited by 2
THE ROLE OF GENERATIVE ARTIFICIAL INTELLIGENCE IN PROGRAMMER TRAINING: OPPORTUNITIES AND CHALLENGES
Hadzhikoleva, S., Rachovski, T., Hadzhikolev, E. N., & Ivanov, I. (2025) · Environment Technology Resources Proceedings of the International Scientific and Practical Conference · doi:10.17770/etr2025vol2.8608
Zhijie, Z., Deng, Y., Qi, H., & Wang, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5351223
Tran, C. T., & Pham, V.-H. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-98164-7_19
Exploring the quality and effectiveness of AI-generated feedback in introductory programming
Qian, Y., Liu, M., & Zhu, L. (2026) · Humanities and Social Sciences Communications · doi:10.1057/s41599-026-07920-7
Integration of AI Code-Writing Assistants in IT Higher Education
Käver, A., Leoste, J., Marmor, K., & Rodríguez-Piñero, P. T. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-07989-3_4 · Cited by 1
Adane, P., Deshpande, N., Dhamecha, A., Telrandhe, O., Kesharwani, S., & Dhawan, V. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-8901-9_22
Enhancing Student Engagement Using PoE and GitHub in Assessment Strategy in a HEI
Joseph, M. K., Jahoor, F., & Rakgogo, T. (2025) · Springer eBooks · doi:10.1007/978-981-95-0687-3_3
AI Chatbots vs. Traditional Search: A Comparative Study on Student Information Retrieval
Pasquarelli, L., Koutcheme, C., & Hellas, A. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328491 · Cited by 1
Survey of Generative Artificial Intelligence Powered Human-Data Interfaces in Big Data Systems
Gao, Y., Roy, A., Chodnekar, V., Solur, C., & Pan, G. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402173
Integrating AI Literacy, Secure Coding, and Critical Evaluation
Atul Kumar Dnyaan Prasad Global University, Namazov, G., Nadira, A., Rizamat, S., Bobur, M., Mohichekhra, K., & Ozoda, A. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-8450-4.ch008 · Cited by 1
PACGBI: A CI/CD Pipeline for LLM-assisted Code Generation of Web Front End
Sarschar, M., Zhang, G., & Nowak, A. (2025) · World Symposium on Software Engineering (WSSE) · doi:10.1145/3779657.3779672
Perceived Importance of Cognitive Skills Among Computing Students in the Era of AI
Rani, N., Cenko, E., & Castro, L. M. C. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--60449
Engineering Students' Usage and Perceptions of GitHub Copilot in Open-Source Projects
Rani, N., Munnangi, J. R., Spangler, A. M., & Honeycutt, D. R. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--59352
Schema-Based Cognition, Embodied Cognition and Social Cognition: The Impact of AI on STEM Education
Soundarajan, N. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-21147-7_20
Adoption and Usability of AI Tools in Bangladesh: Opportunities and Challenges
Hossain, D. T., & Sakib, N. (2025) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit68739.2025.11491700
A Knowledge-Based Method for Designing Knowledge Querying Systems in Computer Programming Education
Huynh, T. T., Mai, T. T., & Do, N. V. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0071-9_21
Carinan, C. J. M., Lucero, H. R., Navarette, M. N., Ortiz, K. U., Paroni, J. K. S., & Duyag, C. B. C. (2026) · International Journal of Latest Technology in Engineering Management & Applied Science · doi:10.51583/ijltemas.2026.150500182
Learning Programming with Generative AI: A Pre-University Pilot Study
Yong, S.-T., Chin, L.-Y., Matthews, R., & Ramayah, B. (2026) · International Conference on Information and Education Technology (ICIET) · doi:10.1109/iciet69664.2026.11561564
DualCodeDetect: Zero-Shot LLM-Generated Code Detection via Dual-Channel Perturbation
Li, Z., Shang, X., Fu, Z., Guo, S., Zhang, W., Yu, N., & Chen, K. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808166
Towards Introductory Programming with ChatGPT-Assisted MTL-Based Program Visualization
Nyamawe, A. S. (2026) · Journal of Computer Languages · doi:10.1016/j.cola.2026.101425
Embracing large language models in education: insights and reflections from the Caribbean context
Ildephonce, I., Hunte, A. A., & Rolle-Greenidge, G. (2026) · Caribbean Educational Research Journal · doi:10.1108/cerj-04-2026-0009
Radar Algorithm Generation System Based on Domain-Adaptive Retrieval-Augmented Generation
Hu, L., Jia, Y., Zhao, B., Xie, M., Li, Y., & Dong, K. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11582809
Karamousalidis, N., & Kefalas, P. (2026) · SN Computer Science · doi:10.1007/s42979-026-05242-8
A Study on Code Clone Lifecycles in Pull Requests Created by AI Agents
Uchoa, I., de Sousa, D. M., Chuvas, H., Paixão, M. H. E., Ragkhitwetsagul, C., & Lima, T. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793608
Secure Banking System with Voice Commands Supported by Agent Navigator for Disabled Customers
Urayabee, R. K., Kwrsheed, A. A., & Abbas, T. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-28307-8_21
Does ChatGPT Consistently Return Secure Cryptographic API Code?
Le, T., Liu, G., & Yue, C. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-4677-0_12
Performance of Large Language Models in Automatic Generation of Logic Programs
Karamousalidis, N., Kefalas, P., & Stamatopoulou, I. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-24513-7_16
Teaching Assistants as Assessors: Deepening the Narrative in Grading Practices
Ahmed, F., Bouali, N., & Gerhold, M. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-21997-8_6
Tadimalla, S. Y., & Maher, M. L. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--60777
Tran, T. P. D., & Nguyen, T. T. V. (2026) · International Conference on Education and Training Technologies (ICETT) · doi:10.1109/icett70124.2026.11666258
Hey, ChatGPT, Look at My Work: Using Conversational AI in Requirements Engineering Education
Badihi, S., Tegegn, M., Riddell, E., Czarnecki, K., & Rubin, J. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773112
Software Development using Low-Code Platforms or AI-Assistants: A Case Study-based Comparison
Hofmann, L. S., Wieber, P., & Taentzer, G. (2026) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1145/3822455.3830306
Beyond Coding: How Students Use Generative AI Across the Full Software Development Lifecycle
Matti, S., Szendrei, R., & Pintér, B. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-32732-1_14
Chat-Fusion: An Efficient Batch Inference System for Cloud LLMs Serving
Yu, J., Wang, L., Wang, M., Li, H., Han, Y., Li, X., Li, H., Zhang, K., & Wang, Y. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3734339
Generative AI for Building Industry: Applications, Challenges, and Future Directions
Le, A. T., Ahn, Y., & Nguyen, A. T. (2026) · Lecture notes in civil engineering · doi:10.1007/978-981-95-6770-6_90
Denny, P. C., Leinonen, J., Hellas, A., Sarsa, S., Liut, M., & Khosravi, H. (2026) · Western Canadian Conference on Computing Education · doi:10.1145/3828792.3828801
Explanation First Prompting for Reviewing AI Programming Answers in Undergraduate Courses
Raiter, O. К. (2026) · International Conference on Advanced Computer Information Technologies (ACIT) · doi:10.1109/acit69831.2026.11700007
Masi, A. D., Wang, C., & Moccozet, L. (2026) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3829807.3829904
Ouh, E. L., Tan, K. W., Lo, S. L., Gan, B. K. S., & Zhang, Y. (2026) · ACM Transactions on Computing Education · doi:10.1145/3858331
LLM-Assisted Code Generation and Reuse: An Exploratory Study on Contextual Information
Reinhartz-Berger, I., Gal-Oz, N., Litvak, M., & Morad, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-40763-4_16
"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv
“Wermelinger [106] evaluated the capabilities of Copilot in the spirit of Finnie-Ainsley et al. [42] who evaluated Codex and found it to still miss the mark more often than not.”
Method
Metadata
- Authors
- Wermelinger, M.
- Affiliations
- The Open University, Milton Keynes, United Kingdom
- ORCID
- Wermelinger, Michel
- Venue
- Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2023-03-02
- Pages
- 172-178
- OpenAlex
- W4324138978
- Cited by
- 191
- References
- 10
- Citations by year
- 2026: 35 · 2025: 65 · 2024: 51 · 2023: 40
- FWCI
- 57.69
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- Open Research Online (The Open University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- GitHub Copilot, OpenAI Codex, large language models, programming education, introductory programming, academic integrity, code generation, code explanation, test case generation, bug fixes, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
