Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Open accessAbstract
This article explores the natural language generation capabilities of large language models with application to the production of two types of learning resources common in programming courses. Using OpenAI Codex as the large language model, we create programming exercises (including sample solutions and test cases) and code explanations, assessing these qualitatively and quantitatively. Our results suggest that the majority of the automatically generated content is both novel and sensible, and in some cases ready to use as is. When creating exercises we find that it is remarkably easy to influence both the programming concepts and the contextual themes they contain, simply by supplying keywords as input to the model. Our analysis suggests that there is significant value in massive generative machine learning models as a tool for instructors, although there remains a need for some oversight to ensure the quality of the generated content before it is delivered to students. We further discuss the implications of OpenAI Codex and similar tools for introductory programming education and highlight future research streams that have the potential to improve the quality of the educational experience for both teachers and students alike.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The OpenAI Codex model can provide a variety of programming tasks together with the correct solution, automated tests to verify the student's solutions, and additional code explanations.
“Using few-shot learning, [40] showed that the OpenAI Codex model is able to provide a variety of programming tasks together with the correct solution, automated tests to verify the student's solutions, and additional code explanations.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [40]
Section 3.2.2 describes using examples to prompt Codex to generate exercises with sample solutions and automated tests, and the conclusion reports promising code explanations. But the paper does not establish that the solutions are consistently correct or that the generated tests reliably verify students’ solutions: Section 4.1 reports matching sample solutions for 76.7% of assessed exercises, and its test analysis found that only 51 of 165 generated solutions passed their generated tests. (LLM, conf=0.88)
Experimental — accuracy not yet measuredRecent studies in learning programming found the correctness of Codex to be around 70%-80%, meaning about 20-30% of Copilot suggestions would be incorrect.
“recent studies in the context of learning programming have found the correctness of Codex (the LLM that powers Copilot) to be around 70%-80% [34, 75], meaning that in about 20-30% of cases, the suggestion by Copilot would be incorrect.”
From “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers (2024) · cites this paper as [75]
Section 4.1 Programming Exercises (page unavailable) reports that 76.7% of the evaluated Codex-generated exercises had a sample solution matching the exercise description, which supports a roughly 70–80% figure for that specific measure. Section 2.4 identifies Codex as powering Copilot, but the study did not measure the correctness of Copilot suggestions; its exercise result cannot establish that 20–30% of those suggestions are incorrect. (LLM, conf=0.86)
Experimental — accuracy not yet measuredFor introductory programming courses, Codex could generate sensible and novel exercises for students along with an appropriate sample solution around three out of four times.
“Additionally, for introductory programming courses, the state-of-the-art LLMs, Codex, could generate sensible and novel exercises for students along with an appropriate sample solution (around three out of four times)”
From Practical and ethical challenges of large language models in education: A systematic scoping review (2024) · cites this paper as Sarsa et al., 2022
Section 4.1 reports that, among 120 evaluated exercises, 75.0% were sensible, 81.8% were novel, and 76.7% had a matching sample solution. The Conclusion says a majority met these criteria, but the copy does not report that around three out of four exercises met all three criteria together; the cited percentages measure them separately. (LLM, conf=0.83)
Experimental — accuracy not yet measuredLLMs have demonstrated remarkable effectiveness in generating novel language artifacts such as course readings, code explanations, model solutions, feedback, and responses to help requests.
“This has been inspired by many other instances where LLMs have already demonstrated remarkable effectiveness in generating novel language artifacts. These include course readings [22], code explanations [23, 28, 29, 41], model solutions [9, 35, 42], feedback [34], or responses to help requests [18, 26, 46, 47].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [41]
These LLMs can write functionally correct code, with capabilities in solving introductory programming tasks and algorithmic challenges.
“These LLMs can write functionally correct code, with studies showing capabilities in solving introductory programming tasks [17, 18] and algorithmic challenges [19])”
From Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022) · cites this paper as [18]
Codex could generate accurate code explanation with 67% accuracy.
“and accurate code explanation (67% accuracy) (Sarsa et al., 2022).”
From Practical and ethical challenges of large language models in education: A systematic scoping review (2024) · cites this paper as Sarsa et al., 2022
AI tools can be used to produce high-quality learning resources, including programming exercises, explanations of code, and on-demand feedback to students.
“On the other hand, recent work has shown that AI tools can be used to produce high-quality learning resources, including programming exercises [32], explanations of code [24,28], and even on-demand feedback to students [27].”
From Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study (2024) · cites this paper as [32]
AI tools could be used to generate code examples and code explanations for code reading skills.
“Considering code reading skills, AI tools could be used to generate code examples [6] and code explanations [32].”
From Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study (2024) · cites this paper as [32]
From the teachers' perspective, AI tools can be used to generate not only examples but complete programming assignments with model solutions and test cases, thus reducing their work and supporting the production of personalized exercises for students.
“From the teachers' perspective, AI tools can be used to generate not only examples but complete programming assignments with model solutions and test cases [32], thus reducing their work and supporting the production of personalized exercises for students.”
From Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study (2024) · cites this paper as [32]
Instructors can benefit as these models can rapidly generate new and personalised teaching materials and programming assignments.
“Instructors can also benefit as these models can rapidly generate new and personalised teaching materials and programming assignments [75,174].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [174]
The exercises generated by LLMs were novel and sensible, although the authors cautioned that the quality might not be good enough to provide the LLM-generated exercises directly to students.
“In addition, the exercises generated by LLMs were novel and sensible, although the authors cautioned that the quality might not be good enough to provide the LLM-generated exercises directly to students.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [174]
LLMs can be coaxed into generating exercises with prescribed themes and programming concepts.
“One of the main findings in both papers was that LLMs can be coaxed into generating exercises with prescribed themes (such as basketball or cooking) and programming concepts (such as loops or conditionals).”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [174]
Codex seems most proficient at crafting line-by-line code explanations, as opposed to higher level summaries of the code.
“This is supported by the findings of Sarsa et al. [174] who noted that Codex seems most proficient at crafting line-by-line code explanations, as opposed to higher level summaries of the code.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [174]
Teachers who use generative AI to assist in the creation of learning resources may unintentionally produce exercises that are underspecified or that contain incorrect reference solutions or inadequate/incorrect test cases.
“Teachers who use generative AI to assist in the creation of learning resources may unintentionally produce exercises that are underspecified or that contain incorrect reference solutions or inadequate/incorrect test cases [174].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [174]
LLMs can ease the burden often felt by teaching assistants and could be a first line of help for students.
“This usecase exemplifies the potential for LLMs to ease the burden often felt by teaching assistants, and could be a first line of help for students [174].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [174]
LLMs can be useful in generating several kinds of learning activities, including novel variations of programming assignments.
“LLMs can be useful in generating several kinds of learning activities, including novel variations of programming assignments [174].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [174]
LLMs can generate good explanations of code.
“LLMs can also generate good explanations of code [94,123,137,174].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [174]
Sarsa et al. found that most problems generated by Codex were novel and sensible, and that the generated code explanations were generally correct and thorough.
“They found that most of the problems generated by Codex were novel and sensible, and that the generated code explanations were generally correct and thorough.”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [36]
Tools like Codex can generate new programming assignments and code explanations when provided examples.
“Tools like Codex are able to generate new programming assignments [36] and code explanations [29] when provided examples.”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(1978) · Medical Entomology and Zoology · Cited by 53,488 · see citing works
Self-efficacy: Toward a unifying theory of behavioral change
Bandura, A. (1977) · Psychological Review · doi:10.1037/0033-295x.84.2.191 · Cited by 43,925 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
The role of deliberate practice in the acquisition of expert performance
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993) · Psychological Review · doi:10.1037/0033-295x.100.3.363 · Cited by 9,564 · see citing works
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · see citing works
Peer Instruction: Ten years of experience and results
Crouch, C. H., & Mazur, E. (2001) · American Journal of Physics · doi:10.1119/1.1374249 · Cited by 2,682 · see citing works
Signature pedagogies in the professions
Shulman, L. S. (2005) · Daedalus · doi:10.1162/0011526054622015 · Cited by 2,036 · see citing works
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013 · see citing works
An Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,750 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Evaluating the Quality of Learning: The Solo Taxonomy : Structure of the Observed Learning Outcome
Biggs, J., & Collis, K. F. (1977) · Cited by 1,087 · see citing works
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · see citing works
Empirical Studies of Programming Knowledge
Soloway, E. M., & Ehrlich, K. (1984) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1984.5010283 · Cited by 879 · see citing works
Strengthening the case for pair programming
Williams, L. A., Kessler, R. R., Cunningham, W., & Jeffries, R. (2000) · IEEE Software · doi:10.1109/52.854064 · Cited by 779 · see citing works
Learning to program = learning to construct mechanisms and explanations
Soloway, E. (1986) · CACM · doi:10.1145/6592.6594 · Cited by 615 · see citing works
Review of recent systems for automatic assessment of programming assignments
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930480 · Cited by 561 · see citing works
A Survey of Automated Assessment Approaches for Programming Assignments
Ala-Mutka, K. (2005) · Computer Science Education · doi:10.1080/08993400500150747 · Cited by 550 · see citing works
Shute, V. J. (2007) · ETS Research Report Series · doi:10.1002/j.2333-8504.2007.tb02053.x · Cited by 467 · see citing works
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · see citing works
Self-citationA systematic literature review of automated feedback generation for programming exercises
Keuning, H., Jeuring, J., & Heeren, B. (2019) · ACM Transactions on Computing Education · doi:10.1145/3231711 · Cited by 370 · see citing works
Large pre-trained language models contain human-like biases of what is right and wrong to do
Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A., & Kersting, K. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00458-8 · Cited by 291 · see citing works
Automated Assessment in Computer Science Education: A State-of-the-Art Review
Paiva, J. C., Leal, J. P., & Figueira, Á. (2022) · ACM Transactions on Computing Education · doi:10.1145/3513140 · Cited by 275 · see citing works
Debugging: a review of the literature from an educational perspective
McCauley, R. A., Fitzgerald, S., Lewandowski, G., Murphy, L., Simon, B., Thomas, L., & Zander, C. (2008) · Computer Science Education · doi:10.1080/08993400802114581 · Cited by 269 · see citing works
Not seeing the forest for the trees
Lister, R., Simon, B., Thompson, E., Whalley, J. L., & Prasad, C. (2006) · ACM SIGCSE Bulletin · doi:10.1145/1140123.1140157 · Cited by 266 · see citing works
Conditions of Learning in Novice Programmers
Perkins, D. N., Hancock, C., Hobbs, R., Martin, F., & Simmons, R. (1986) · Journal of Educational Computing Research · doi:10.2190/gujt-jcbj-q6qu-q9pl · Cited by 262 · see citing works
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243 · see citing works
Vihavainen, A., Airaksinen, J., & Watson, C. (2014) · doi:10.1145/2632320.2632349 · Cited by 224 · see citing works
Pair programming in education: a literature review
Hanks, B., Fitzgerald, S., McCauley, R. A., Murphy, L., & Zander, C. (2011) · Computer Science Education · doi:10.1080/08993408.2011.579808 · Cited by 207 · see citing works
Whalley, J. L., Lister, R., Thompson, E., Clear, T., Robbins, P., Kumar, P. K. A., & Prasad, C. (2006) · Tuwhera (Auckland University of Technology) · Cited by 193 · see citing works
A closer look at tracing, explaining and code writing skills in the novice programmer
Venables, A. T., Tan, G., & Lister, R. (2009) · doi:10.1145/1584322.1584336 · Cited by 188 · see citing works
Drori, I., Zhang, S., Shuttleworth, R., Tang, L., Lu, A., Ke, E., Liu, K., Chen, L., Tran, S., Cheng, N., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2123433119 · Cited by 187 · see citing works
A multi-national study of reading and tracing skills in novice programmers
Lister, R., Adams, E. S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, J. E., Sanders, K., Seppälä, O., et al. (2004) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1044550.1041673 · Cited by 186 · see citing works
Extreme apprenticeship method in teaching programming for beginners
Vihavainen, A., Paksula, M., & Luukkainen, M. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953196 · Cited by 177 · see citing works
A surprising effect of feedback on learning
Vollmeyer, R., & Rheinberg, F. (2005) · Learning and Instruction · doi:10.1016/j.learninstruc.2005.08.001 · Cited by 176 · see citing works
Expectancy-Value Theory and Its Relevance for Student Motivation and Learning
Rosenzweig, E. Q., Wigfield, A., & Eccles, J. S. (2019) · Cambridge University Press eBooks · doi:10.1017/9781316823279.026 · Cited by 154 · see citing works
A theory of instruction for introductory programming skills
Xie, B., Loksa, D., Nelson, G. L., Davidson, M. J., Dong, D., Kwik, H., Tan, A. H., Hwa, L., Li, M., & Ko, A. J. (2019) · Computer Science Education · doi:10.1080/08993408.2019.1565235 · Cited by 148 · see citing works
Williams, J. J., Kim, J., Rafferty, A. N., Maldonado, S., Gajos, K. Z., Lasecki, W. S., & Heffernan, N. T. (2016) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2876034.2876042 · Cited by 147 · see citing works
Lobb, R., & Harlow, J. (2016) · ACM Inroads · doi:10.1145/2810041 · Cited by 120 · see citing works
The recurring rainfall problem
Fisler, K. (2014) · doi:10.1145/2632320.2632346 · Cited by 112 · see citing works
Going SOLO to assess novice programmers
Sheard, J., Carbone, A., Lister, R., et al. (2008) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1384271.1384328 · Cited by 102 · see citing works
Do we know how difficult the rainfall problem is?
Seppälä, O., Ihantola, P., Isohanni, E., Sorva, J., & Vihavainen, A. (2015) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2828959.2828963 · Cited by 93 · see citing works
Cognitive Load Theory in Computing Education Research: A Review
Duran, R., Zavgorodniaia, A., & Sorva, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3483843 · Cited by 92 · see citing works
Sanders, K., Ahmadzadeh, M., Clear, T., Edwards, S. H., Goldweber, M., Johnson, C., Lister, R., McCartney, R., Patitsas, E., & Spacco, J. W. (2013) · doi:10.1145/2543882.2543885 · Cited by 91 · see citing works
Denny, P. C., Luxton-Reilly, A., Tempero, E. D., & Hendrickx, J. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953299 · Cited by 89 · see citing works
Self-citationAn Explicit Strategy to Scaffold Novice Program Tracing
Xie, B., Nelson, G. L., & Ko, A. J. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159527 · Cited by 82 · see citing works
Ullah, Z., Lajis, A., Jamjoom, M. M., Altalhi, A. H., Al‐Ghamdi, A., & Saleem, F. (2018) · Computer Applications in Engineering Education · doi:10.1002/cae.21974 · Cited by 80 · see citing works
Common logic errors made by novice programmers
Ettles, A., Luxton-Reilly, A., & Denny, P. C. (2018) · Australasian Computing Education Conference · doi:10.1145/3160489.3160493 · Cited by 70 · see citing works
Self-citationA Review of Research on Parsons Problems
Du, Y., Luxton-Reilly, A., & Denny, P. C. (2020) · Australasian Computing Education Conference · doi:10.1145/3373165.3373187 · Cited by 69 · see citing works
Self-citationPlagiarism in Programming Assessments: A Systematic Review
Albluwi, I. (2020) · ACM Transactions on Computing Education · doi:10.1145/3371156 · Cited by 63 · see citing works
Lister, R., Fidge, C., & Teague, D. (2009) · ACM SIGCSE Bulletin · doi:10.1145/1595496.1562930 · Cited by 62 · see citing works
Ability to 'explain in plain english' linked to proficiency in computer-based programming
Murphy, L., Fitzgerald, S., Lister, R., & McCauley, R. A. (2012) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2361276.2361299 · Cited by 57 · see citing works
Plagiarism in Take-home Exams: Help-seeking, Collaboration, and Systematic Cheating
Hellas, A., Leinonen, J., & Ihantola, P. (2017) · Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3059009.3059065 · Cited by 50 · see citing works
Self-citationSTOP THE (AUTOGRADER) INSANITY: Regression Penalties to Deter Autograder Overreliance
Baniassad, E., Zamprogno, L., Hall, B., & Holmes, R. (2021) · Proceedings of the 52nd ACM Technical Symposium on Computer Science Education · doi:10.1145/3408877.3432430 · Cited by 48 · see citing works
Identification of programmers from typing patterns
Longi, K., Leinonen, J., Nygren, H., Salmi, J., Arto Klami, A., & Vihavainen, A. (2015) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2828959.2828960 · Cited by 44 · see citing works
Self-citationAutograding "Explain in Plain English" questions using NLP
Fowler, M., Chen, B., Azad, S., West, M., & Zilles, C. B. (2021) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3408877.3432539 · Cited by 41 · see citing works
Developing Course-Level Learning Goals for Basic Data Structures in CS2
Porter, L., Zingaro, D., Lee, C. B., et al. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159457 · Cited by 40 · see citing works
Weekly Programs in a CS1 Class: Experiences with Auto-graded Many-small Programs (MSP)
Allen, J. M., Vahid, F., Downey, K., & Edgcomb, A. (2020) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--31231 · Cited by 40 · see citing works
Sanders, K., Boustedt, J., Eckerdal, A., McCartney, R., & Zander, C. (2017) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3105726.3106192 · Cited by 34 · see citing works
Edwards, J., Ditton, J., Trninić, D., Swanson, H., Sullivan, S., & Mano, C. D. (2020) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3372782.3406259 · Cited by 34 · see citing works
Yan, L., Hu, A., & Piech, C. (2019) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3287324.3287483 · Cited by 32 · see citing works
A Validated Scoring Rubric for Explain-in-Plain-English Questions
Chen, B., Azad, S., Haldar, R., West, M., & Zilles, C. B. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366879 · Cited by 32 · see citing works
A Comparison of Immediate and Scheduled Feedback in Introductory Programming Projects
Leinonen, J., Denny, P., & Whalley, J. (2022) · Proceedings of the 53rd ACM Technical Symposium on Computer Science Education · doi:10.1145/3478431.3499372 · Cited by 32 · see citing works
Self-citationFalkner, K. E. K., & Sheard, J. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.016 · Cited by 27 · see citing works
Developing a common format for sharing programming assignments
Edwards, S. H., Börstler, J., Cassel, L. N., et al. (2008) · ACM SIGCSE Bulletin · doi:10.1145/1473195.1473240 · Cited by 26 · see citing works
Ciniselli, M., Pascarella, L., & Bavota, G. (2022) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3524842.3528440 · Cited by 26 · see citing works
Crowdsourcing programming assignments with CrowdSorcerer
Pirttinen, N., Kangas, V., Nikkarinen, I., et al. (2018) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3197091.3197117 · Cited by 25 · see citing works
Wrenn, J., Krishnamurthi, S., & Fisler, K. (2018) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3230977.3230999 · Cited by 25 · see citing works
Empirical studies of programming knowledge
Soloway, E., & Ehrlich, K. (1989) · doi:10.1145/75722.75734 · Cited by 25 · see citing works
Singh, A., Brooks, C. A., Lin, Y., & Li, W. (2021) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3430895.3460142 · Cited by 23 · see citing works
Denny, P. C., Cukierman, D., & Bhaskar, J. (2015) · doi:10.1145/2828959.2828967 · Cited by 22 · see citing works
Self-citationExploring the Effects of Contextualized Problem Descriptions on Problem Solving
Leinonen, J., Denny, P. C., & Whalley, J. L. (2021) · Australasian Computing Education Conference · doi:10.1145/3441636.3442302 · Cited by 20 · see citing works
Self-citationTyping Patterns and Authentication in Practical Programming Exams
Leinonen, J., Longi, K., Arto Klami, A., Ahadi, A., & Vihavainen, A. (2016) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2899415.2899472 · Cited by 19 · see citing works
Self-citationExamining a Student-Generated Question Activity Using Random Topic Assignment
Denny, P. C., Tempero, E. D., Garbett, D., et al. (2017) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3059009.3059033 · Cited by 19 · see citing works
Self-citationLet’s Ask Students About Their Programs, Automatically
Lehtinen, T., Santos, A. L., & Sorva, J. (2021) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc52881.2021.00054 · Cited by 19 · see citing works
On the cognitive development of the novice programmer
Lister, R. (2020) · doi:10.1145/3442481.3442498 · Cited by 18 · see citing works
Nip, T., Gunter, E. L., Herman, G. L., Morphew, J., & West, M. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159500 · Cited by 17 · see citing works
Harnessing the Wisdom of the Classes
Saarinen, S., Krishnamurthi, S., Fisler, K., et al. (2019) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3287324.3287504 · Cited by 17 · see citing works
Crowdsourcing Content Creation for SQL Practice
Leinonen, J., Pirttinen, N., & Hellas, A. (2020) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3341525.3387385 · Cited by 17 · see citing works
Self-citationHassan, M. R., & Zilles, C. B. (2021) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3446871.3469765 · Cited by 17 · see citing works
Examining Zero-Shot Vulnerability Repair with Large Language Models
Pearce, H., Tan, B., Ahmad, B., Karri, R., & Dolan-Gavitt, B. (2021) · arXiv · doi:10.48550/arxiv.2112.02125 · Cited by 17 · see citing works
Pop Quiz! Can a Large Language Model Help With Reverse Engineering?
Pearce, H., Tan, B., Krishnamurthy, P., Khorrami, F., Karri, R., & Dolan-Gavitt, B. (2022) · arXiv · doi:10.48550/arxiv.2202.01142 · Cited by 16 · see citing works
A Study of Pair Programming Enjoyment and Attendance using Study Motivation and Strategy Metrics
Aarne, O., Peltola, P., Leinonen, J., & Hellas, A. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159493 · Cited by 14 · see citing works
Self-citationMaking instructions for others
Phillips, R. D., Lockton, D., Baurley, S. L., et al. (2013) · interactions · doi:10.1145/2505290 · Cited by 13 · see citing works
Hovemeyer, D., Hertz, M., Denny, P. C., et al. (2013) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2445196.2445451 · Cited by 12 · see citing works
Self-citationQi, R., & Fossati, D. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366939 · Cited by 12 · see citing works
Misconception-Based Peer Feedback: A Pedagogical Technique for Reducing Misconceptions
Kennedy, C., Lawson, A., Feaster, Y., & Kraemer, E. T. (2020) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3341525.3387392 · Cited by 10 · see citing works
An Experience Using On-Computer Programming Questions During Exams
Stephenson, B. (2018) · Western Canadian Conference on Computing Education · doi:10.1145/3209635.3209639 · Cited by 9 · see citing works
Can Novice Programmers Write C Functions?
Izu, C., & Dinh, P. (2018) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale.2018.8615375 · Cited by 9 · see citing works
Non-restricted Access to Model Solutions: A Good Idea?
Nygren, H., Leinonen, J., & Hellas, A. (2019) · Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3304221.3319768 · Cited by 7 · see citing works
Self-citationExperimenting with Model Solutions as a Support Mechanism
Nygren, H., Leinonen, J., Pirttinen, N., Leinonen, A., & Hellas, A. (2019) · Proceedings of the 1st UK & Ireland Computing Education Research Conference on - UKICER · doi:10.1145/3351287.3351288 · Cited by 4 · see citing works
Self-citationPachler, N., Evans, M., Redondo, A., et al. (2013) · Routledge eBooks · doi:10.4324/9780203181928-14 · Cited by 1 · see citing works
Ciniselli, M., Pascarella, L., & Bavota, G. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.06894
Flagged first, then by citations
Research works printed but not linked: 5
- [9]9. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in neural information processing systems. 1877-1901.
- [15]15. Paul Denny, Andrew Luxton-Reilly, Ewan Tempero, and Jacob Hendrickx. 2011. Codewrite: supporting student-driven practice of java. In Proceedings of the 42nd ACM technical symposium on Computer science education. 471-476.
- [32]32. John Hattie and Helen Timperley. 2007. The power of feedback. Review of educational research 77, 1 (2007), 81-112.
- [41]41. Juho Kim. 2015. Learnersourcing: improving learning with collective learner activity. Ph. D. Dissertation. Massachusetts Institute of Technology.
- [94]94. Lisa Yan, Annie Hu, and Chris Piech. 2019. Pensieve: Feedback on Coding Process for Novices. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE '19). Association for Computing Machinery, New York, NY, USA, 253-259.
Cited by
Sorted by citations
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · see citing works
Practical and ethical challenges of large language models in education: A systematic scoping review
Yan, L., Sha, L., Zhao, L., Li, Y., Martinez‐Maldonado, R., Chen, G., Li, X., Jin, Y., & Gašević, D. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13370 · Cited by 935 · see citing works
“Additionally, for introductory programming courses, the state-of-the-art LLMs, Codex, could generate sensible and novel exercises for students along with an appropriate sample solution (around three out of four times) and accurate code explanation (67% accuracy) (Sarsa et al., 2022).”
BackgroundThe promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543 · see citing works
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401 · see citing works
Self-citationThe Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · see citing works
Self-citationA Comprehensive Review on Generative AI for Education
Mittal, U., Sai, S., Chamola, V., & Sangwan, D. (2024) · IEEE Access · doi:10.1109/access.2024.3468368 · Cited by 383 · see citing works
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366 · see citing works
“Additionally, these tools could support educators in curriculum development by generating programming exercises and explanations to solutions [79].”
InfluentialUsing an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 345 · see citing works
CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models
Lemieux, C., Inala, J. P., Lahiri, S. K., & Sen, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00085 · Cited by 306 · see citing works
Computing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316 · see citing works
Self-citationDenny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287 · see citing works
Self-citationGenerative AI for Customizable Learning Experiences
Pesovski, I., Santos, R., Henriques, R., & Trajkovik, V. (2024) · Sustainability · doi:10.3390/su16073034 · Cited by 274 · see citing works
“In a more technical field such as programming, Sarsa et al. [54] found LLMs to be effective not only in generating novel introductory programming exercises but also in providing comprehensive and accurate explanations of most of the lines of code that make up the solution.”
BackgroundPrather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256 · see citing works
Self-citationComparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588785 · Cited by 210 · see citing works
Self-citationUsing GitHub Copilot to Solve Simple Programming Problems
Wermelinger, M. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569830 · Cited by 191 · see citing works
MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569785 · Cited by 195 · see citing works
Self-citationThe Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196 · see citing works
Self-citationCodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192 · see citing works
Self-citationMy AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E. A., Prather, J., & Becker, B. A. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576134 · Cited by 186 · see citing works
Self-citationPrompt Problems: A New Programming Exercise for the Generative AI Era
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630909 · Cited by 187 · see citing works
Self-citationSavelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141 · see citing works
Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145 · see citing works
Self-citationGenerative AI in Computing Education: Perspectives of Students and Instructors
Zastudil, C., Rogalska, M., Kapp, C., Vaughn, J., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343467 · Cited by 136 · see citing works
A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132 · see citing works
Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors
Phung, T., Pădurean, V.-A., Cambronero, J., Gulwani, S., Kohn, T., Majumdar, R., Singla, A., & Soares, G. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2 · doi:10.1145/3568812.3603476 · Cited by 112 · see citing works
Natural language processing in the era of large language models
Zubiaga, A. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1350306 · Cited by 81 · see citing works
Detecting LLM-Generated Text in Computing Education: Comparative Study for ChatGPT Cases
Orenstrakh, M. S., Karnalim, O., Suárez, C. A., & Liut, M. (2024) · 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00027 · Cited by 79 · see citing works
Investigating the Potential of GPT-3 in Providing Feedback for Programming Assessments
Balse, R., Valaboju, B., Singhal, S., Warriem, J. M., & Prasad, P. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588852 · Cited by 63 · see citing works
Lyu, W., Zhang, S., Chung, T., Sun, Y., & Zhang, Y. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100383 · Cited by 68 · see citing works
“On the one hand, a growing body of literature has documented the potential of integrating GenAI into higher education, such as generating course materials [34, 99] and assisting students’ programming tasks [74, 93].”
Generating Multiple Choice Questions for Computing Courses Using Large Language Models
Tran, A., Angelikas, K., Rama, E., Okechukwu, C., Smith, D. H., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342898 · Cited by 60 · see citing works
Reeves, B., Sarsa, S., Prather, J., Denny, P., Becker, B. A., Hellas, A., Kimmel, B., Powell, G., & Leinonen, J. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588805 · Cited by 56 · see citing works
Self-citationExploring ChatGPT’s impact on post-secondary education: A qualitative study
Rajabi, P., Taghipour, P., Cukierman, D., & Doleck, T. (2023) · Proceedings of the 25th Western Canadian Conference on Computing Education · doi:10.1145/3593342.3593360 · Cited by 57 · see citing works
Students’ Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programming
Zhang, Z., Dong, Z., Shi, Y., Price, T., Matsuda, N., & Xu, D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i21.30372 · Cited by 54 · see citing works
Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study
Sheard, J., Denny, P., Hellas, A., Leinonen, J., Malmi, L., & Simon (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630880 · Cited by 54 · see citing works
Self-citationBudhiraja, R., Joshi, I., Challa, J. S., Akolekar, H. D., & Kumar, D. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636257 · Cited by 52 · see citing works
ChatGPT in Ukrainian Education: Problems and Prospects
Fiialka, S., Kornieva, Z., & Honcharuk, T. (2023) · International Journal of Emerging Technologies in Learning (iJET) · doi:10.3991/ijet.v18i17.42215 · Cited by 45 · see citing works
Jordan, M., Ly, K., & Soosai Raj, A. G. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630897 · Cited by 39 · see citing works
Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants
Sandoval, G., Pearce, H., Nys, T., Karri, R., Garg, S., & Dolan-Gavitt, B. (2022) · arXiv · doi:10.48550/arxiv.2208.09727 · Cited by 52 · see citing works
Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics
Pathak, A., Gandhi, R., Uttam, V., Ramamoorthy, A., Ghosh, P., Jindal, A. R., Verma, S., Mittal, A., Ased, A., Khatri, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.23989 · Cited by 30
Artificial intelligence in computer programming education: A systematic literature review
Manorat, P., Tuarob, S., & Pongpaichet, S. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100403 · Cited by 27 · see citing works
Socratic Questioning of Novice Debuggers: A Benchmark Dataset and Preliminary Evaluations
Al-Hossami, E., Bunescu, R., Teehan, R., Powell, L., Mahajan, K., & Dorodchi, M. (2023) · Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023) · doi:10.18653/v1/2023.bea-1.57 · Cited by 23 · see citing works
Prather, J., Reeves, B. N., Denny, P., Leinonen, J., MacNeil, S., Luxton-Reilly, A., Orvalho, J., Alipour, A., Alfageeh, A., Amarouche, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.12800 · Cited by 20 · see citing works
Self-citationVIVID: Human-AI Collaborative Authoring of Vicarious Dialogues from Lecture Videos
Choi, S., Lee, H., Lee, Y., & Kim, J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642867 · Cited by 17 · see citing works
Robosourcing Educational Resources -- Leveraging Large Language Models for Learnersourcing
Denny, P., Sarsa, S., Hellas, A., & Leinonen, J. (2022) · arXiv · doi:10.48550/arxiv.2211.04715 · Cited by 21 · see citing works
Self-citationChang, C. I., Choi, W. C., & Choi, I. C. (2024) · Proceedings of the 2024 7th Artificial Intelligence and Cloud Computing Conference · doi:10.1145/3719384.3719439 · Cited by 11 · see citing works
“Similarly, Sarsa et al. [11] found that while OpenAI Codex’s explanations covered most of the code, they contained inaccuracies, particularly in explaining comparison and branching conditionals.”
Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3
Bellettini, C., Lodi, M., Lonati, V., Monga, M., & Morpurgo, A. (2023) · Proceedings of the 15th International Conference on Computer Supported Education · doi:10.5220/0012007500003470 · Cited by 10 · see citing works
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B., Kimmel, B., Wright, J., & Briggs, B. (2024) · arXiv · Cited by 196 · see citing works
Self-citationLi, Y., & Bao, Z. (2025) · Proceedings of the 2025 2nd International Conference on Artificial Intelligence and Future Education · doi:10.1145/3785987.3786027 · Cited by 1 · see citing works
Zhu, S., & Liu, Z. (2026) · Proceedings of the 2026 International Conference on Big Data and Informatization Education · doi:10.1145/3806980.3806985
Mapeamento Sistemático sobre o Uso de IAs Generativas no Ensino de Programação
Ferreira de Sousa, J., & Silveira, F. R. D. V. (2026) · Revista Eletrônica de Iniciação Científica em Computação · doi:10.5753/reic.2026.7169
"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv
Self-citation
- Authors
- Sarsa, S., Denny, P., Hellas, A., & Leinonen, J.
- Affiliations
- Aalto University (FI)
- University of Auckland (NZ)
- Venue
- Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1
- Publisher
- ACM
- Published
- 2022-08-03
- Pages
- 27-43
- OpenAlex
- W4283705032
- Semantic Scholar
- 0d08ffccc982781e310bb184397bbe64b9aef157
- Cited by
- 497
- References
- 94
- Citations by year
- 2026: 85 · 2025: 151 · 2024: 170 · 2023: 88 · 2022: 3
- FWCI
- 134.39
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Aaltodoc (Aalto University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- code explanation, large language models, OpenAI Codex, natural language generation, programming education, introductory programming, test case generation, sample solutions, instructor tools
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
