Evaluating the Performance of Code Generation Models for Solving Parsons Problems With Small Prompt Variations
Open accessAbstract
The recent emergence of code generation tools powered by large language models has attracted wide attention. Models such as OpenAI Codex can take natural language problem descriptions as input and generate highly accurate source code solutions, with potentially significant implications for computing education. Given the many complexities that students face when learning to write code, they may quickly become reliant on such tools without properly understanding the underlying concepts. One popular approach for scaffolding the code writing process is to use Parsons problems, which present solution lines of code in a scrambled order. These remove the complexities of low-level syntax, and allow students to focus on algorithmic and design-level problem solving. It is unclear how well code generation models can be applied to solve Parsons problems, given the mechanics of these models and prior evidence that they underperform when problems include specific restrictions. In this paper, we explore the performance of the Codex model for solving Parsons problems over various prompt variations. Using a corpus of Parsons problems we sourced from the computing education literature, we find that Codex successfully reorders the problem blocks about half of the time, a much lower rate of success when compared to prior work on more free-form programming tasks. Regarding prompts, we find that small variations in prompting have a noticeable effect on model performance, although the effect is not as pronounced as between different problems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The 'many small programs' approach may have become obsolete given how easily most LLMs can now solve introductory computing problems with simple prompts.
“However, this approach may have become obsolete given how easily most LLMs can now solve introductory computing problems with simple prompts [85,86,167,175].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [167]
For other tasks, such as answering multiple-choice questions, Parsons problems, and computational thinking tasks, the performance of LLMs is currently not as great.
“For other tasks, such as answering MCQs [176], Parsons problems [167] and computational thinking tasks [39], the performance of LLMs is currently not as great.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [167]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 2× · see citing works
“This increased when the user utilised “prompt engineering” – modifying the prompt given to the model with the aim of achieving better performance [28].”
BackgroundMethodAutomatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · see citing works
Self-citationThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · see citing works
Self-citationGitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417 · see citing works
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417 · see citing works
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401 · see citing works
Self-citationUsing Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235 · see citing works
Self-citationEvaluating a new exam question
Denny, P. C., Luxton-Reilly, A., & Simon, B. (2008) · doi:10.1145/1404520.1404532 · Cited by 203 · see citing works
Self-citationGenerating Diverse Code Explanations using the GPT-3 Large Language Model
MacNeil, S., Tran, A., Mogil, D., Bernstein, S., Ross, E., & Huang, Z. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 2 · doi:10.1145/3501709.3544280 · Cited by 200 · see citing works
MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569785 · Cited by 195 · see citing works
Self-citationMy AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E. A., Prather, J., & Becker, B. A. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576134 · Cited by 186 · see citing works
Self-citationSubgoals Help Students Solve Parsons Problems
Morrison, B. B., Margulieux, L. E., Ericson, B. J., & Guzdial, M. (2016) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2839509.2844617 · Cited by 126 · see citing works
Solving parsons problems versus fixing and writing code
Ericson, B. J., Margulieux, L. E., & Rick, J. (2017) · Proceedings of the 17th Koli Calling International Conference on Computing Education Research · doi:10.1145/3141880.3141895 · Cited by 121 · see citing works
The recurring rainfall problem
Fisler, K. (2014) · doi:10.1145/2632320.2632346 · Cited by 112 · see citing works
How do students solve parsons programming problems?
Helminen, J., Ihantola, P., Karavirta, V., & Malmi, L. (2012) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2361276.2361300 · Cited by 99 · see citing works
Welsh, M. (2022) · Communications of the ACM · doi:10.1145/3570220 · Cited by 95 · see citing works
Do we know how difficult the rainfall problem is?
Seppälä, O., Ihantola, P., Isohanni, E., Sorva, J., & Vihavainen, A. (2015) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2828959.2828963 · Cited by 93 · see citing works
Evaluating the Efficiency and Effectiveness of Adaptive Parsons Problems
Ericson, B. J., Foley, J. D., & Rick, J. (2018) · Proceedings of the 2018 ACM Conference on International Computing Education Research · doi:10.1145/3230977.3231000 · Cited by 70 · see citing works
A Review of Research on Parsons Problems
Du, Y., Luxton-Reilly, A., & Denny, P. C. (2020) · Australasian Computing Education Conference · doi:10.1145/3373165.3373187 · Cited by 69 · see citing works
Self-citationImproving Instruction of Programming Patterns with Faded Parsons Problems
Weinman, N., Fox, A., & Hearst, M. A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445228 · Cited by 62 · see citing works
Distractors in Parsons Problems Decrease Learning Efficiency for Young Novice Programmers
Harms, K. J., Chen, J., & Kelleher, C. L. (2016) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2960310.2960314 · Cited by 60 · see citing works
Ericson, B. J., Denny, P. C., Prather, J. E., Duran, R., Hellas, A., Leinonen, J., Miller, C. S., Morrison, B. B., Pearce, J. L., & Rodger, S. H. (2022) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3571785.3574127 · Cited by 59 · see citing works
Self-citationTwo-Dimensional Parson’s Puzzles: The Concept, Tools, and First Observations
Ihantola, P., & Karavirta, V. (2011) · Journal of Information Technology Education Innovations in Practice · doi:10.28945/1394 · Cited by 56 · see citing works
A mobile learning application for parsons problems with automatic feedback
Karavirta, V., Helminen, J., & Ihantola, P. (2012) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2401796.2401798 · Cited by 51 · see citing works
Automatically Generating CS Learning Materials with Large Language Models
MacNeil, S., Tran, A., Leinonen, J., Denny, P. C., Kim, J., Hellas, A., Bernstein, S., & Sarsa, S. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545947.3569630 · Cited by 48 · see citing works
Self-citationKumar, A. N. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159576 · Cited by 35 · see citing works
Haynes-Magyar, C., & Ericson, B. J. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445292 · Cited by 32 · see citing works
Using Adaptive Parsons Problems to Scaffold Write-Code Problems
Hou, X., Ericson, B. J., & Wang, X. (2022) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3501385.3543977 · Cited by 32 · see citing works
When Wrong is Right: The Instructional Power of Multiple Conceptions
Margulieux, L. E., Denny, P. C., Cunningham, K. I., Deutsch, M., & Shapiro, B. R. (2021) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3446871.3469750 · Cited by 28 · see citing works
Self-citationCheng, N., & Harrington, B. (2017) · doi:10.1145/3017680.3017704 · Cited by 26 · see citing works
Robosourcing Educational Resources -- Leveraging Large Language Models for Learnersourcing
Denny, P., Sarsa, S., Hellas, A., & Leinonen, J. (2022) · arXiv · doi:10.48550/arxiv.2211.04715 · Cited by 21 · see citing works
Self-citationAdaptive Problem Selection in a Mobile Python Tutor
Fabic, G. V. F., Mitrović, A., & Neshatian, K. (2018) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3213586.3225235 · Cited by 18 · see citing works
Exploring Challenging Variations of Parsons Problems
Weinman, N., Fox, A., & Hearst, M. A. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3372639 · Cited by 14 · see citing works
Denny, P. C., Kumar, V., & Giacaman, N. (2022) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · doi:10.48550/arxiv.2210.15157 · Cited by 13 · see citing works
Self-citationHelping Students Solve Parsons Puzzles Better
Kumar, A. N. (2019) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3304221.3319735 · Cited by 12 · see citing works
Adaptive Parsons Problems as Active Learning Activities During Lecture
Ericson, B. J., & Haynes-Magyar, C. (2022) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3502718.3524808 · Cited by 10 · see citing works
Scaffolding the Design Process using Parsons Problems
Garcia, R., Falkner, K. E., & Vivian, R. (2018) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3279720.3279746 · Cited by 8 · see citing works
The Impact of Solving Adaptive Parsons Problems with Common and Uncommon Solutions
Haynes-Magyar, C., & Ericson, B. J. (2022) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3564721.3564736 · Cited by 8 · see citing works
Regex Parsons: Using Horizontal Parsons Problems to Scaffold Learning Regex
Wu, Z., Ericson, B. J., & Brooks, C. A. (2021) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3488042.3489968 · Cited by 5 · see citing works
Automatically Generating CS Learning Materials with Large Language ModelsPreprint
MacNeil, S., Tran, A., Leinonen, J., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.05113 · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 5
- Barbara J. Ericson, Mark J. Guzdial, and Briana B. Morrison. 2015. Analysis of Interactive Features Designed to Enhance Learning in an Ebook. In Proc. of the 11th Annual Int. Conf. on Int. Computing Education Research. ACM, 169-178.
- Dale Parsons and Patricia Haden. 2006. Parson's Programming Puzzles: A Fun and Effective Learning Tool for First Programming Courses. In Proc. of the 8th Australasian Conf. on Computing Education -Volume 52 (ACE '06). Australian Computer Society, Inc., AUS, 157-163.
- James Prather, John Homer, Paul Denny, Brett Becker, John Marsden, et al. 2022. Scaffolding Task Planning Using Abstract Parsons Problems. In Proc. of the 2022 World Conf. on Computers in Education (WCCE '22). 1-10.
- Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Conf. on Human Factors in Computing Systems Extended Abstracts. ACM, NY NY, USA, 1-7.
- Rui Zhi, Min Chi, Tiffany Barnes, and Thomas W. Price. 2019. Evaluating the Effectiveness of Parsons Problems for Block-Based Programming. In Proc. of the 2019 ACM Conf. on Int. Computing Education Research. ACM, NY, NY, USA, 51-59.
Cited by
Sorted by citations
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · see citing works
Self-citationComputing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316 · see citing works
“Newer models can also solve other types of programming exercises, like Parsons Problems, with decent accuracy that is likely to only increase over time [31].”
BackgroundSelf-citationKazemitabaar, M., Ye, R., Wang, X.-H., Henley, A. Z., Denny, P. C., Craig, M., & Grossman, T. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642773 · Cited by 280 · see citing works
Self-citationPrather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256 · see citing works
“evaluated the ability of GPT-3 to solve Parsons Problems and found that the nature of generative AI tools as next-token-predictors means that they often struggle to only use the lines of code provided to them in the prompt to solve the puzzle [90].”
BackgroundSelf-citationPrompt Problems: A New Programming Exercise for the Generative AI Era
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630909 · Cited by 187 · see citing works
Self-citationPrather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709614 · Cited by 97 · see citing works
Self-citationGenerative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factors
Ghimire, A., Pather, J., & Edwards, J. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892891 · Cited by 88 · see citing works
Joshi, I., Budhiraja, R., Dev, H., Kadia, J., Ataullah, M. O., Mitra, S., Akolekar, H. D., & Kumar, D. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630803 · Cited by 80 · see citing works
Budhiraja, R., Joshi, I., Challa, J. S., Akolekar, H. D., & Kumar, D. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636257 · Cited by 52 · see citing works
Jordan, M., Ly, K., & Soosai Raj, A. G. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630897 · Cited by 39 · see citing works
Kleinig, O., Gao, C., Kovoor, J. G., Gupta, A. K., Bacchi, S., & Chan, W. O. (2023) · Eye · doi:10.1038/s41433-023-02772-w · Cited by 37 · see citing works
Computing education using generative artificial intelligence tools: A systematic literature review
Agbo, F. J., Olivia, C., Oguibe, G. F., Sanusi, I. T., & Sani, G. (2025) · Computers and Education Open · doi:10.1016/j.caeo.2025.100266 · Cited by 48 · see citing works
Pirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 37 · see citing works
Hou, I., Man, O., Mettille, S., Gutierrez, S., Angelikas, K., & MacNeil, S. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636247 · Cited by 31 · see citing works
Mahon, J., Namee, B. M., & Becker, B. A. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653602 · Cited by 30 · see citing works
Self-citationEnhancing Programming Error Messages in Real Time with Generative AI
Kimmel, B., Geisert, A. L., Yaro, L., Gipson, B., Hotchkiss, R., Osae-Asante, S. K., Vaught, H., Wininger, G., & Yamaguchi, C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3647967 · Cited by 29 · see citing works
Self-citationZamfirescu-Pereira, J. D., Qi, L., Hartmann, B., DeNero, J., & Norouzi, N. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701864 · Cited by 28 · see citing works
Literature Review on the Integration of Generative AI in Programming Education
Nathaniel, J., Oyelere, S. S., Suhonen, J. T., & Tedre, M. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00524-3 · Cited by 40 · see citing works
Generative AI and Generative Pre-Trained Transformer Applications
Rosário, A. T. (2024) · Advances in media, entertainment and the arts (AMEA) book series · doi:10.4018/979-8-3693-1950-5.ch003 · Cited by 26 · see citing works
Smith, D. H., Denny, P. C., & Fowler, M. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3662039 · Cited by 22 · see citing works
Self-citationPereira, A. F., & Mello, R. F. (2025) · IEEE Access · doi:10.1109/access.2025.3584060 · Cited by 22 · see citing works
Prather, J., Reeves, B. N., Denny, P., Leinonen, J., MacNeil, S., Luxton-Reilly, A., Orvalho, J., Alipour, A., Alfageeh, A., Amarouche, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.12800 · Cited by 20 · see citing works
Self-citationPrather, J. E., Reeves, B. N., Denny, P. C., Leinonen, J., MacNeil, S., Luxton-Reilly, A., Orvalho, J. G., Alipour, M. A., Alfageeh, A., Amarouche, T., et al. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716649 · Cited by 20 · see citing works
Self-citationGenerative AI in Computer Science Education: Challenges and Opportunities
Franklin, D., Denny, P., Gonzalez-Maldonado, D. A., & Tran, M. (2025) · doi:10.1017/9781009581738 · Cited by 20 · see citing works
Self-citationFeng, T. H., Denny, P., Wuensche, B., Luxton-Reilly, A., & Hooper, S. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636263 · Cited by 16 · see citing works
Self-citationMahon, J., Namee, B. M., & Becker, B. A. (2023) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3610969.3610982 · Cited by 15 · see citing works
Self-citationAnalyzing LLM Usage in an Advanced Computing Class in India
Arora, U., Garg, A., Gupta, A., Jain, S., Mehta, R. L., Oberoi, R., Prachi, Raina, A., Saini, M., Sharma, S., et al. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716657 · Cited by 15 · see citing works
Smith, D. H., & Zilles, C. B. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653582 · Cited by 14 · see citing works
A High-Quality Generation Approach for Educational Programming Projects Using LLM
Song, T., Zhang, H., & Xiao, Y. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3499751 · Cited by 14 · see citing works
Cipriano, B. P., & Alves, P. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640052 · Cited by 13 · see citing works
Multi-Institutional Multi-National Studies of Parsons Problems
Ericson, B. J., Pearce, J. L., Rodger, S. H., Csizmadia, A. P., Garcia, R., Gutiérrez, F. J., Liaskos, K. B., Padiyath, A., Scott, M., Smith, D. H., et al. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3623762.3633498 · Cited by 13 · see citing works
Iterative Student Program Planning using Transformer-Driven Feedback
Rivera, E., Steinmaurer, A., Fisler, K., & Krishnamurthi, S. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653607 · Cited by 12 · see citing works
An Investigation of the Drivers of Novice Programmers' Intentions to Use Web Search and GenAI
Skripchuk, J., Bacher, J. T., & Price, T. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671112 · Cited by 12 · see citing works
Towards Integrating Emerging AI Applications in SE Education
Vierhauser, M., Groher, I., Antensteiner, T., & Sauerwein, C. (2024) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet62301.2024.10663045 · Cited by 11 · see citing works
A Bibliometric Exposition and Review on Leveraging LLMs for Programming Education
Amos, J. P., Amodu, O. A., Mahmood, R. A. R., AbdulQudus, A. B., Zakaria, A. F., Iyanda, A. R., Bukar, U. A., & Hanapi, Z. M. (2025) · IEEE Access · doi:10.1109/access.2025.3554627 · Cited by 13 · see citing works
Song, T., Tian, Q., Xiao, Y., & Liu, S. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-0730-0_28 · Cited by 10 · see citing works
Solving Proof Block Problems Using Large Language Models
Poulsen, S., Sarsa, S., Prather, J. E., Leinonen, J., Becker, B. A., Hellas, A., Denny, P. C., & Reeves, B. N. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630928 · Cited by 12 · see citing works
Self-citationKohn, T. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70043 · Cited by 13 · see citing works
Feng, H., Denny, P. C., Wuensche, B. C., Luxton-Reilly, A., & Whalley, J. L. (2024) · doi:10.1145/3680533.3697064 · Cited by 10 · see citing works
Self-citationThe Robots are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00658 · Cited by 9 · see citing works
Self-citationLi, N., Broner, S., Kim, Y., Mizuo, K., Sauder, E., To, C., Wang, A., Gila, O., & Shindler, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701946 · Cited by 8 · see citing works
Fostering Responsible AI Use Through Negative Expertise: A Contextualized Autocompletion Quiz
MacNeil, S., Prather, J. E., Nabid, R. A., Gutierrez, S., Carvalho, S., Shrestha, S., Denny, P. C., Reeves, B. N., Leinonen, J., & Rossetti, R. L. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729067 · Cited by 7 · see citing works
Self-citationCognitive Apprenticeship and Artificial Intelligence Coding Assistants
Poitras, E., Crane, B. G. C., Dempsey, D., Bragg, T. A., Siegel, A. A., & Lin, M. P.-C. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-1066-3.ch013 · Cited by 7 · see citing works
Ali, M., Ma, M., Muneeb, M., & Wong, G. K. W. (2025) · Metrics · doi:10.3390/metrics2040023 · Cited by 6 · see citing works
Haque, S., & Hundhausen, C. D. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744231 · Cited by 5 · see citing works
Guiding ChatGPT for Better Code Generation: An Empirical Study
Liu, C., Bao, X., Zhang, H., Zhang, N., Hu, H., Zhang, X., & Yan, M. L. (2024) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner60148.2024.00018 · Cited by 6 · see citing works
Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming
Chen, R., Jiang, S., Shen, J., Moon, A., & Wei, L. (2025) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1109/aiware69974.2025.00016 · Cited by 3 · see citing works
ÖZGÜL, E. (2026) · Kastamonu Eğitim Dergisi · doi:10.24106/kefdergi.1878122 · Cited by 3 · see citing works
Servín, C., Hawthorne, E. K., Postner, L., Tang, C., & Tucker, C. S. (2023) · Conference on Information Technology Education · doi:10.1145/3585059.3611441 · Cited by 3 · see citing works
Leveling up Learning: Serious Games for Computing Education - Long-Term Opportunities and Risks
Wuensche, B. C., Hooper, S., Whalley, J. L., Straand, I. J., Denny, P. C., Crow, T. V. H., Lange-Nawka, D. C., Luxton-Reilly, A., & Thompson, S. E. R. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716655 · Cited by 3 · see citing works
Self-citationQuality Assessment of Python Tests Generated by Large Language Models
Alves, V. A., Bezerra, C. I. M., Machado, I., Rocha, L., Virgínio, T., & Silva, P. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756964 · Cited by 2 · see citing works
Programmers' Views on IDE Compilation Mechanisms
Karvelas, I., Dillane, J., & Becker, B. A. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617915
Self-citationIMF-MF: Interactive moment localization with adaptive multimodal fusion and self-attention
Singh, P., Kushwaha, A. K. S., & Varshney, N. (2024) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-233071
Aquino, B., Brito, A., Tavares, C. S., Boechat, D., & Batisteli, J. P. O. (2025) · doi:10.5753/vem.2025.14303
ChatGPT and Generative Artificial Intelligence: A Bibliometric Review of the Research
Wamba, S. F., & Guthrie, C. H. (2025) · Lecture notes in networks and systems · doi:10.1007/978-981-96-4142-0_17
Özyürek, H., & Türen, U. (2026) · SN Computer Science · doi:10.1007/s42979-026-04883-z
Ortega, M. F., de Carvalho, L. S. G., Fernandes, D., & de Holanda, M. T. (2026) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2026.27469
"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv
“evaluated the ability of GPT-3 to solve Parsons Problems and found that the nature of generative AI tools as next-token-predictors means that they often struggle to only use the lines of code provided to them in the prompt to solve the puzzle [90].”
BackgroundSelf-citation
- Authors
- Reeves, B., Sarsa, S., Prather, J., Denny, P., Becker, B. A., Hellas, A., Kimmel, B., Powell, G., & Leinonen, J.
- Affiliations
- Abilene Christian University (US)
- Aalto University (FI)
- University of Auckland (NZ)
- University College Dublin (IE)
- ORCID
- Venue
- Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2023-06-29
- Pages
- 299-305
- OpenAlex
- W4382652775
- Cited by
- 56
- References
- 37
- Citations by year
- 2026: 3 · 2025: 19 · 2024: 26 · 2023: 8
- FWCI
- 16.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) (publishedVersion, cc-by)
- Aaltodoc (Aalto University) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Keywords
- Parsons problems, code generation, large language models, OpenAI Codex, prompt variation, programming education, prompt engineering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
