Prompt Problems: A New Programming Exercise for the Generative AI Era
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) are revolutionizing the field of computing education with their powerful code-generating capabilities. Traditional pedagogical practices have focused on code writing tasks, but there is now a shift in importance towards reading, comprehending and evaluating LLM-generated code. Alongside this shift, an important new skill is emerging -- the ability to solve programming tasks by constructing good prompts for code-generating models. In this work we introduce a new type of programming exercise to hone this nascent skill: 'Prompt Problems'. Prompt Problems are designed to help students learn how to write effective prompts for AI code generators. A student solves a Prompt Problem by crafting a natural language prompt which, when provided as input to an LLM, outputs code that successfully solves a specified programming task. We also present a new web-based tool called Promptly which hosts a repository of Prompt Problems and supports the automated evaluation of prompt-generated code. We deploy Promptly in one CS1 and one CS2 course and describe our experiences, which include student perceptions of this new type of activity and their interactions with the tool. We find that students are enthusiastic about Prompt Problems, and appreciate how the problems engage their computational thinking skills and expose them to new programming constructs. We discuss ideas for the future development of new variations of Prompt Problems, and the need to carefully study their integration into classroom practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Further, novice learners of programming may become overwhelmed and confused by generated code, which could add to the high levels of frustration that are common in introductory programming courses [71,188].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [71]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although LLMs can automate routine programming tasks, their limitations become apparent when addressing nuanced or ill-structured problems (Zhang et al., 2023;Denny et al. 2024).”
From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Denny et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Explicit instruction on how to critically assess, refine, and debug AI-generated code fosters the development of essential competencies -ranging from algorithmic reasoning to software testing (Denny et al., 2024;Kendon et al., 2023).”
From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Denny et al., 2024
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Although generative systems are stateless in that they do not retain context across interactions unless explicitly prompted [107,108]”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [107]
Experimental — accuracy not yet measuredIn the citing paper
“Indeed, some recent papers found in the second phase snowballing focused on introducing tools around LLMs, such as CodeHelp [127] and Promptly [71].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [71]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838 · cited 1×
“White et al. present a prompt structuring framework for constructing prompts that can be applied across problem domains, and demonstrate how prompts can be constructed from patterns [30].”
BackgroundAutomatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
Self-citationThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
Self-citationBecker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401 · cited 1×
“In addition to the opportunities they present, educators have voiced concerns around the potential misuse of these models for plagiarism, and over-reliance on AI-generated code by beginners [3], leading to a possible erosion of traditional coding skills [9].”
BackgroundSelf-citationThe Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · cited 1×
“The advent of large language models (LLMs) that can generate code is having a rapid and significant impact on computing education practice, particularly at the introductory level [24].”
BackgroundSelf-citationStudying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287
Self-citationMalinka, K., Peresíni, M., Firc, A., Hujnák, O., & Janus, F. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588827 · Cited by 287
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
Self-citationLau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241
Research works printed but not linked: 2
- Leo Porter and Daniel Zingaro. 2023. Learn AI-Assisted Python Programming: With Github Copilot and ChatGPT. Manning, Shelter Island, NY.
- 10.1007/978--3-031--11647--6_127 · doi:10.1007/978--3-031--11647--6_127
Cited by
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
Self-citationBengesi, S., El-Sayed, H., Sarker, M. K., Houkpati, Y., Irungu, J., & Oladunni, T. (2024) · IEEE Access · doi:10.1109/access.2024.3397775 · Cited by 362
“This prompt is crucial in determining the generated output, making prompt engineering a vital skill [142].”
BackgroundComputing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“Depending on the prompt, the resulting programs used varied programmatic structures, while ultimately favoring expected methods for each problem variation (i.e., for-loops for processing lists, and while-loops for processing standard input).”
BackgroundSelf-citationAI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · arXiv · doi:10.48550/arxiv.2505.10468 · Cited by 295
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
“• Prompt Dependency and Statelessness: Although generative systems are stateless in that they do not retain context across interactions unless explicitly prompted [107,108], recent advancements like GPT-4.”
Computer Science Curricula 2023
Kumar, A. N., Raj, R. K., Aly, S. G., Anderson, M. D., Becker, B. A., Blumenthal, R. L., Eaton, E., Epstein, S. L., Goldweber, M., Jalote, P., et al. (2024) · ACM eBooks · doi:10.1145/3664191 · Cited by 223
Self-citationThe Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196
“Many computing education tools that utilize generative AI and LLMs have recently emerged such as CodeHelp [32], CodeAid [27], and Promptly [13].”
Self-citationAI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186
Sun, L., & Zhou, L. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241277937 · Cited by 129
“In natural science courses, small class sizes facilitate detailed experimental design and data analysis, so code generation can be used to assist students with programming tasks and scienti fi c computation (Denny et al., 2024).”
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709614 · Cited by 209
“…[86, 133], WorkedGen [57], LLM Hint Factory [158], Tutor Kai [52], CodeTutor [90], GPTeach [96], Charlie the Coding Cow [103], KOGI [78], NetLogo Chat [24], Promptly [33, 34], IPSSC [85], CS50 Duck [87], Gilt [100], CodeTailor [50], and a variety of others that lacked names [76, 97, 101, 156].”
Self-citationLarge Language Models in Computer Science Education: A Systematic Literature Review
Raihan, M. N., Siddiq, M. L., Santos, J. C. S., & Zampieri, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701863 · Cited by 219
Generative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factors
Ghimire, A., Pather, J., & Edwards, J. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892891 · Cited by 88
“New programming problem types that utilize LLMs, such as Prompt Problems, are also beginning to emerge [23].”
AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744217 · Cited by 163
“…literacy that emerged that often directly address generative AI tools: effective usage of generative AI tools for post-secondary students (e.g. [20, 26, 107, 110]), incorporating AI tools into K-12 AI literacy curricula (e.g. [50, 97, 113]), and increased mentions of the indirectly beneficial…”
InfluentialAI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · arXiv · doi:10.48550/arxiv.2503.00079 · Cited by 73
“…literacy that emerged that often directly address generative AI tools: effective usage of generative AI tools for post-secondary students (e.g. [20, 26, 107, 110]), incorporating AI tools into K-12 AI literacy curricula (e.g. [50, 97, 113]), and increased mentions of the indirectly beneficial…”
InfluentialMargulieux, L. E., Prather, J., Reeves, B. N., Becker, B. A., Cetin Uzun, G., Loksa, D., Leinonen, J., & Denny, P. (2024) · Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3649217.3653621 · Cited by 59
“Emerging pedagogical techniques that integrate LLMs, such as “Prompt Problems” are attempting to do this by helping students to learn how to use AI while still learning to code [13].”
BackgroundSelf-citationAriza, J. Á., Restrepo, M. B., & Hérnandez, C. H. (2025) · IEEE Access · doi:10.1109/access.2025.3541424 · Cited by 100
Koutcheme, C., Dainese, N., Sarsa, S., Hellas, A., Leinonen, J., & Denny, P. (2024) · Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3649217.3653612 · Cited by 47
“Although there will always be situations where students choose to use closed source models to seek feedback or answer questions, a wide range of new educational tools powered by LLMs are being developed and integrated into computing classrooms [4, 11, 20].”
BackgroundSelf-citationDenny, P. C., Smith, D. H., Fowler, M., Prather, J. E., Becker, B. A., & Leinonen, J. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653587 · Cited by 86
Self-citationSoftware engineering education in the era of conversational AI: current trends and future directions
Sengul, C., Neykova, R., & Destefanis, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1436350 · Cited by 83
Student-AI Interaction: A Case Study of CS1 students
Amoozadeh, M., Nam, D., Prol, D., Alfageeh, A., Prather, J. E., Hilton, M. L., Ragavan, S. S., & Alipour, M. A. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699567 · Cited by 90
Self-citationKohen-Vacs, D., Usher, M., & Jansen, M. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00496-4 · Cited by 39
“Although LLMs can automate routine programming tasks, their limitations become apparent when addressing nuanced or ill-structured problems (Zhang et al., 2023; Denny et al. 2024).”
What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
Ma, Q., Peng, W., Yang, C., Shen, H., Koedinger, K. R., & Wu, T. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3731756 · Cited by 111
“If after 2 rounds of questioning the user hasn’t provided certain details, use the data available to generate the itinerary.”
Kazemitabaar, M., Huang, O., Suh, S., Henley, A. Z., & Grossman, T. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712104 · Cited by 92
“Ultimately, our results highlight the most effective technique: guiding learners through the step-by-step problem-solving process, where they engage in an interactive dialog with the AI, prompting what needs to be done at each stage before the corresponding code is revealed.”
Pirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 65
Humble, N. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00494-x · Cited by 57
“…on GAI technologies (Albdrani & Al-Shargabi, 2023; Amoozadeh et al., 2024; Becker et al., 2023; Budhiraja et al., 2024; Cambaz & Zhang, 2024; Denny et al., 2024c; Fernandez et al., 2024; Finnie-Ansley et al., 2023; Groothuijsen et al., 2024; Gumina et al., 2023; Hellas et al., 2023;…”
InfluentialEnhancing Programming Error Messages in Real Time with Generative AI
Kimmel, B., Geisert, A. L., Yaro, L., Gipson, B., Hotchkiss, R., Osae-Asante, S. K., Vaught, H., Wininger, G., & Yamaguchi, C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3647967 · Cited by 44
“They can even be used to create entirely new types of problems for students to solve, such as Prompt Problems, which ask students to write the prompt that would solve a given problem [4].”
Literature Review on the Integration of Generative AI in Programming Education
Nathaniel, J., Oyelere, S. S., Suhonen, J. T., & Tedre, M. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00524-3 · Cited by 66
“…for engagement, moti - vating learners to abstract key elements, decompose complex problems, and logically deal with each more minor problem (Denny et al., 2024), building confidence and enabling a sense of ownership over the technology, ensuring learners become active participants rather…”
Generative AI and Power Imbalances in Global Education: Frameworks for Bias Mitigation
Nyaaba, M., Wright, A., & Choi, G. L. (2024) · arXiv · doi:10.48550/arxiv.2406.02966 · Cited by 25
Smith, D. H., Denny, P. C., & Fowler, M. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3662039 · Cited by 40
“Denny et al. [13] introduced “Prompt Problems”, an approach to teaching students to prompt LLMs that provides students with a visual representation of input and output, asks the student to infer the operation(s) being performed, generates code based on a description they provide, and uses unit…”
Self-citationAutomating Autograding: Large Language Models as Test Suite Generators for Introductory Programming
Alkafaween, U., Al-Bluwi, I., & Denny, P. C. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13100 · Cited by 35
“They have been applied to generating new programming exercises [39], explaining code [12, 22, 24, 33], providing feedback and improving the readability of programming error messages [23, 33], and powering novel pedagogical approaches [11, 45].”
Self-citationPrather, J. E., Reeves, B. N., Denny, P. C., Leinonen, J., MacNeil, S., Luxton-Reilly, A., Orvalho, J. G., Alipour, M. A., Alfageeh, A., Amarouche, T., et al. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716649 · Cited by 44
“Initial work has shown that students find Prompt Problems engaging and encourage metacognitive reflection [13].”
InfluentialSelf-citationGenerative AI in Computer Science Education: Challenges and Opportunities
Franklin, D., Denny, P., Gonzalez-Maldonado, D. A., & Tran, M. (2025) · doi:10.1017/9781009581738 · Cited by 20
Self-citationPrather, J., Reeves, B. N., Denny, P., Leinonen, J., MacNeil, S., Luxton-Reilly, A., Orvalho, J., Alipour, A., Alfageeh, A., Amarouche, T., et al. (2024) · arXiv · doi:10.48550/arxiv.2412.12800 · Cited by 20
“Initial work has shown that students find Prompt Problems engaging and encourage metacognitive reflection [13].”
InfluentialSelf-citationFeng, H., Luxton-Reilly, A., Wuensche, B. C., & Denny, P. C. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716658 · Cited by 46
“Importantly, these tasks allow students to refine their problem-specification skills, a critical aspect of AI-assisted programming.”
Self-citationLópez‐Pernas, S., Misiejuk, K., Kaliisa, R., & Saqr, M. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3568599 · Cited by 21
“It reinforces their conceptual grasp and improves their ability to communicate theoretical knowledge effectively [34].”
Lyu, W., Wang, Y., Sun, Y., & Zhang, Y. (2025) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3698205.3729544 · Cited by 35
“GenAI demonstrated strong capabilities in coding tasks [15, 28], prompting researchers to investigate its role in CS education [12, 41, 42].”
SunoCaps: A novel dataset of text-prompt based AI-generated music with emotion annotations
Civit, M., Drai-Zerbib, V., Lizcano, D., & Escalona, M. J. (2024) · Data in Brief · doi:10.1016/j.dib.2024.110743 · Cited by 16
“This section deals predominantly with prompt alignment [ 10 ] problems.”
BackgroundSantos, P., Cantalejo, K. U., & Moreno, V. (2024) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro60963.2024.10569779 · Cited by 17
“The resolution of Prompt Problems has been implemented in Computer Science courses to help students learn how to write effective prompts for AI code generators, shifting the importance of code writing tasks towards code reading, comprehension, and evaluation of LLM-generated code [21].”
BackgroundNot the Silver Bullet: LLM-enhanced Programming Error Messages are Ineffective in Practice
Santos, E. A., & Becker, B. A. (2024) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3689535.3689554 · Cited by 28
“LLMs have even enabled brand new pedagogical approaches [10, 28].”
Self-citationAnalysis of Generative AI Policies in Computing Course Syllabi
Ali, A., Collier, A. H., Dewan, U., McDonald, N., & Johri, A. (2025) · Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3641554.3701823 · Cited by 14
“There is also some indication that the shift from writing to "reading, comprehending and evaluating" code will make teaching prompt engineering a vital skill but this will require thoughtful and deliberate integration into computing classrooms [9].”
The Role of Generative AI in Software Student CollaborAItion
Kiesler, N., Smith, J., Leinonen, J., Fox, A., MacNeil, S., & Ihantola, P. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729040 · Cited by 23
“We have seen such initial efforts related to prompt engineering by tool creators [12, 13] and educational researchers trying to elicit feedback [2, 30, 35].”
Self-citationGong, X., Xu, W., & Qiao, A. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-025-00552-y · Cited by 21
“For example, over-reliance on GAI may weaken students’ problem-solving abilities, hinder the development of meta-cognitive skills essential for good programming (Denny et al., 2024a, b), and reduce the satisfaction derived from solving problems independently (Kazemitabaar et al., 2023).”
Integrating Natural Language Prompting Tasks in Introductory Programming Courses
Kerslake, C. P., Denny, P. C., Smith, D. H., Prather, J. E., Leinonen, J., Luxton-Reilly, A., & MacNeil, S. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690125 · Cited by 33
Self-citationBridging Novice Programmers and LLMs with Interactivity
Yeh, T. Y., Tran, K., Gao, G., Yu, T. T., Fong, W. O., & Chen, T.-Y. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701867 · Cited by 28
“CS education work on prompt generation includes [6], in which the authors describe “prompt problems” and an associated tool called Promptly that gives students a visual representation of a problem and then asks students to write prompts that will generate correct code.”
Zhang, L., Jiang, Q., Xiong, W., & Zhao, W. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251333874 · Cited by 15
“If students fail to specify the expected behaviors or functions of their code, ChatGPT should prompt them to describe the desired outcomes or behaviors explicitly before providing substantive assistance (Denny et al., 2024).”
An Investigation of the Drivers of Novice Programmers' Intentions to Use Web Search and GenAI
Skripchuk, J., Bacher, J. T., & Price, T. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671112 · Cited by 22
Stone, I. (2024) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3689535.3689553 · Cited by 22
“…of the selected studies used methods conducive to exploring student perceptions and insights, such as surveys, interviews, focus groups, log data, reflections, [2, 6, 8–10, 14–16, 21– 23, 30–33, 35, 36, 38], this represents a relatively small fraction of the 142 abstracts screened (Figure 1).”
InfluentialA Bibliometric Exposition and Review on Leveraging LLMs for Programming Education
Amos, J. P., Amodu, O. A., Mahmood, R. A. R., AbdulQudus, A. B., Zakaria, A. F., Ìyàndá, A., Bukar, U., & Hanapi, Z. (2025) · IEEE Access · doi:10.1109/access.2025.3554627 · Cited by 13
“The authors [28] introduce Promptly, a web-based tool that provides a repository of prompt problems and facilitates automated evaluation of prompt-generated code.”
InfluentialKohn, T. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70043 · Cited by 15
Automating Personalized Parsons Problems with Customized Contexts and Concepts
Gutierrez, A. D. C., Denny, P. C., & Luxton-Reilly, A. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653568 · Cited by 21
“LLM-powered tools are also entering computing classrooms as digital teaching assistants [25], programming support tools [21], and tools to support the development of skills in prompt creation and engineering [6, 8].”
BackgroundSelf-citationCipriano, B. P., Alves, P., & Denny, P. C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72315-5_6 · Cited by 14
Self-citationFeng, H., Denny, P. C., Wuensche, B. C., Luxton-Reilly, A., & Whalley, J. L. (2024) · doi:10.1145/3680533.3697064 · Cited by 15
“Additionally, LLMs are capable of assisting CS educators by generating educational material [Leinonen et al. 2023; Liffiton et al. 2023; MacNeil et al. 2023], providing many new opportunities [Bernstein et al. 2024; Denny et al. 2024a].”
Self-citationA Pedagogical Framework for Developing Abstraction Skills
Begum, M., Crossley, J., Strömbäck, F., Akrida, E. C., Alpizar-Chacón, I., Evans, A. C., Gross, J. B., Haglund, P., Lonati, V., Satyavolu, C. J., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709613 · Cited by 15
“Denny et al. [23] introduced a novel programming exercise called “Prompt Problems,” designed to teach students how to create effective prompts for AI-based code-generation tools.”
Designing LLM-Resistant Programming Assignments: Insights and Strategies for CS Educators
McDanel, B., & Novak, E. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701872 · Cited by 18
Bridging the Gap: Exploring the Revolutionary Application of GenAI in Language Teaching and Learning
Jadhav, S. (2024) · Journal of Electrical Systems · doi:10.52783/jes.2325 · Cited by 9
Probeable Problems for Beginner-level Programming-with-AI Contests
Pawagi, M., & Kumar, V. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671108 · Cited by 11
“To counteract this mindset, Shin and Kazerouni [43] encourage a “requirements-first” approach to testing.”
Brown, N. C. C., Weill-Tessier, P., Leinonen, J., Denny, P. C., & Kölling, M. (2025) · ACM Transactions on Computing Education · doi:10.1145/3737885 · Cited by 14
Self-citationFostering Responsible AI Use Through Negative Expertise: A Contextualized Autocompletion Quiz
MacNeil, S., Prather, J. E., Nabid, R. A., Gutierrez, S., Carvalho, S., Shrestha, S., Denny, P. C., Reeves, B. N., Leinonen, J., & Rossetti, R. L. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729067 · Cited by 19
Self-citationGensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoning
Liu, M. X., Petridis, S., Tsai, V., Fiannaca, A. J., Olwal, A., Terry, M. A., & Cai, C. J. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712085 · Cited by 11
“…particularly for users without technical expertise in LLMs. Common challenges include struggling with finding the right phrasing for a prompt, selecting appropriate demonstrative examples, experimenting with various hyper-parameters, and evaluating the effectiveness of their prompts [10, 22, 79].”
Leinonen, J., Denny, P. C., Kiljunen, O., MacNeil, S., Sarsa, S., & Hellas, A. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716647 · Cited by 20
“Such systems include, for example, CodeAid [25], CodeHelp [31], and Promptly [11].”
Self-citationPromptology: Enhancing Human-AI Interaction in Large Language ModelsPreprint
Olla, P., Elliott, L., & Abumeeiz, M. (2024) · Preprints.org · doi:10.20944/preprints202408.0334.v1 · Cited by 7
Ortega-Arranz, A., Topali, P., & Molenaar, I. (2025) · Proceedings of the 15th International Learning Analytics and Knowledge Conference · doi:10.1145/3706468.3706533 · Cited by 6
Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12
“Finally (in Section 4), we derive a framework that provides structured guidance to educators and policy makers on how to reason about ethical and societal issues when preparing policy documents related to GenAI use in their institutions.”
Explain in Plain Language Questions with Indic Languages: Drawbacks, Affordances, and Opportunities
Smith, D. H., Kumar, V., & Denny, P. C. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-84391-4_1 · Cited by 16
Self-citationSerafini, R., Yardım, A., & Naiakshina, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713989 · Cited by 15
“These issues have prompted researchers to advocate for intervention strategies, such as task decomposition, guardrails within AI tools, and instructional practices that promote independent learning [4, 8, 14–16, 36, 43, 59, 71].”
Philbin, C. A., & Sentance, S. (2025) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3754508.3754513 · Cited by 6
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B., Kimmel, B., Wright, J., & Briggs, B. (2024) · arXiv · Cited by 196
“Many computing education tools that utilize generative AI and LLMs have recently emerged such as CodeHelp [32], CodeAid [27], and Promptly [13].”
Self-citationIterative Design of a Teaching Assistant Training Program in Computer Science Using the Agile Method
Liu, R., Chen, S., Chen, J., Niu, S., Ma, Y., & Tang, X. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701829 · Cited by 5
Exploring Student Reactions to LLM-Generated Feedback on Explain in Plain English Problems
Kerslake, C. P., Denny, P. C., Smith, D. H., Leinonen, J., MacNeil, S., Luxton-Reilly, A., & Becker, B. A. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701934 · Cited by 18
Self-citationKusam, V. A., Shrestha, S., Kattan, K. A., Maxim, B. R., & Song, Z. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893118 · Cited by 7
“Generative Artificial Intelligence (GAI) represents a cutting-edge field within artificial intelligence that focuses on developing algorithms capable of producing content that mirrors human-like outputs, such as text, images, audio and video [1], [2].”
Haque, S., & Hundhausen, C. D. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744231 · Cited by 13
“To understand students’ concerns about trust [5] and over-reliance [9] as they relate to GenAI assistants, we asked how often students evaluated solutions provided by GenAI assistants, and their perceived ability to provide sufficient prompts [14].”
Developing a framework for addressing ethical challenges in generative AI
Lucas, S., Heinitz, R. M., Becker, S. J., & Charton, J. E. (2025) · Journal of Information Technology Case and Application Research · doi:10.1080/15228053.2025.2558443 · Cited by 8
“These include the expertise to classify the output of a prompt (related to the literacy principle) (Denny et al., 2024; Romero Moreno, 2024), the use of personal data in a prompt (proportionality) (Popowicz-Pazdej, 2023; Wach et al., 2023), awareness of the limitations of the generative AI…”
Qiao, S., Denny, P. C., & Giacaman, N. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716651 · Cited by 15
Self-citationEvaluating ChatGPT to Answer Multi-Modal Exercises in Computer Science Education
Ouh, E. L., Tan, K. W., Lo, S. L., & Gan, B. K. S. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729056 · Cited by 7
Cipriano, B. P., Silva, M. M. D. L., Correia, R., & Alves, P. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699588 · Cited by 5
“The tool’s functionality was defined by a group of 3 computer science (CS) teachers, taking their own experience, as well as scientific reports from related tools (e.g. [4, 7, 11]), when defining the plugin’s requirements.”
de Deus, W. S., Marcolino, A. D. S., Avellar, G. M. N., Oliveira, K. K. D. S., & Barbosa, E. F. (2024) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2024.242283 · Cited by 4
Ramírez-Rueda, R., Benítez–Guerrero, E., Mezura-Godoy, C., & Bárcenas, E. (2024) · Programming and Computer Software · doi:10.1134/s0361768824700737 · Cited by 4
From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions
Alfageeh, A., Zarkouei, S. A. K., Nam, D., Prol, D., Amoozadeh, M., Chattopadhyay, S., Prather, J. E., Denny, P. C., Leinonen, J., Hilton, M. L., et al. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770004 · Cited by 7
“We realize the Prompt2Constraints framework for analyzing students’ prompts in programming tasks in a study by Denny et al. [5].”
InfluentialSelf-citationTowards the Integration of Large Language Models in an Object-Oriented Programming Course
Cipriano, B. P. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649405.3659473 · Cited by 7
Reeves, B. N., Prather, J. E., Denny, P. C., Leinonen, J., MacNeil, S., Luxton-Reilly, A., Nicolajsen, S. M., & Brabrand, C. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770039 · Cited by 10
Self-citationA Review of Large Language Model Evaluation Metrics in Education
Paoprasert, N., Chui, K. T., Yip, T. Y., Liu, J., & Ng, K. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4499-8_33 · Cited by 3
Ruggiano, N., Sahoo, S., Brashear, A., Nwatu, U. L., Brunson, A., Noh, H., Cole, H. S., McKinney, R., Suarez, C. V. F., Brown, E. L., et al. (2026) · JMIR Medical Education · doi:10.2196/83085 · Cited by 5
Nakayama, M., Wan, Y., & Alvino, F. (2024) · Artificial Intelligence for Business (AIxB) · doi:10.1109/aixb62249.2024.00014 · Cited by 5
“Obtaining the optimal code for specific programming tasks is an emerging, important skill, and users need adequate interfaces for efficient interactions with LLMs [17].”
Opportunities and considerations for using artificial intelligence in bioinformatics education
Piccolo, S. R., Nathan, A., Brazas, M. D., Kandpal, M., Miró-Herrans, A. T., Kleinschmit, A. J., McClatchy, S., Mutheiwana, P., Nikolic, D., Gallo, L. I., et al. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbaf169 · Cited by 2
“These learning objectives can be incorporated into bioinformatics training through exercises such as “prompt engi-neering” (Denny et al. 2024).”
PLAID: Supporting Computing Instructors to Identify Domain-Specific Programming Plans at Scale
Jain, Y., Demirtaş, M. A., & Cunningham, K. I. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713832 · Cited by 4
Smith, D. H., Fowler, M., Denny, P. C., & Zilles, C. B. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729045 · Cited by 7
Self-citationPozdniakov, S., Brazil, J. R., Poquet, O., Krusche, S., Berrezueta-Guzmán, J., Sadiq, S. W., & Khosravi, H. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98465-5_37 · Cited by 3
“M2M employs state-of-the-art LLM prompting techniques [11, 18, 12, 13], particularly prompt chaining, noted for superior results [12].”
Li, Z., Zheng, X., & Fu, Q. (2025) · International Conference on Computer Science and Technologies in Education (CSTE) · doi:10.1109/cste64638.2025.11092183 · Cited by 4
“AI tools that generate code assist students with weak syntax skills but may hinder fundamental syntax learning [17].”
Help or Hype? Exploring LLM-based Chatbots in Self-Regulated Learning
Tsang, R., & WOOD, S. (2025) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--56687 · Cited by 3
A Phrase Fill-in-Blank Problem in a Client-Side Web Programming Assistant System
Qi, H., Li, Z., Funabiki, N., Kyaw, H. H. S., & Kao, W.-C. (2025) · Information · doi:10.3390/info16080709 · Cited by 2
Prompt Engineering as an AI Literacy Competence: A Framework for Learners and Educators
Oreški, D., Kišić, A., & Rožman, M. (2025) · International Conference on e-Learning (eLearning) · doi:10.46793/elearning2025.160o · Cited by 5
Vadaparty, A., Smith, D. H., Srinath, S., Padala, M., Alvarado, C., Benario, J. G., Porter, L., & Zingaro, D. (2026) · Australasian Computing Education Conference · doi:10.1145/3786228.3786234 · Cited by 6
From Code Generation to Conceptual Learning: Student Use of LLMs in a Web Programming Course
Isa, H.-Y., Weston, M., Wellyanto, M. R., Karna, I., Talton, J. O., & Kumar, R. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3793207 · Cited by 4
Using a Low-Code Environment to Teach Programming in the Era of LLMs
Potriasaeva, A., Dzialets, K., Golubev, Y., & Birillo, A. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632621.3671429 · Cited by 2
AI in CS Education: Opportunities, Challenges, and Pitfalls to Avoid
Vahid, F. (2024) · ACM Inroads · doi:10.1145/3679205 · Cited by 2
“Some CS educators have begun embracing approaches that use AI actively in learning programing, such as discussed in [5,17].”
Work-in-Progress: Students' Prompting Strategies When Solving an Engineering Design Task
Reeping, D. P., & Shah, A. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893099 · Cited by 4
“We should note that, like previous research [9], [12], these prompts are limited to students from a single institution.”
Fowler, M., Emeka, C., Chen, B., Smith, D. H., West, M., & Zilles, C. B. (2025) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3774752 · Cited by 4
Frydenberg, M., Xu, A., & Xu, J. (2025) · Information Systems Education Journal · doi:10.62273/dvnq9288 · Cited by 1
Work in Progress: Unlocking Code Generation Through Synergistic Prompt Engineering
Ho, K.-H., Georgiades, M., Fan, T.-K. J., Hou, Y., Fong, K. C. K., & Chan, T.-T. (2025) · IEEE World Engineering Education Conference (EDUNINE) · doi:10.1109/edunine62377.2025.10980842 · Cited by 2
Creative Learning Using Generative AI: Empowering Problem-Solving and Programming Skills
Elanwar, R. (2025) · Springer series on cultural computing · doi:10.1007/978-3-031-86551-0_7 · Cited by 2
Metadata
- Authors
- Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N.
- Affiliations
- University of Auckland (NZ)
- Abilene Christian University (US)
- University of Toronto (CA)
- University College Dublin (IE)
- ORCID
- Venue
- Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2024-03-07
- Pages
- 296-302
- OpenAlex
- W4392564669
- Cited by
- 187
- References
- 27
- Citations by year
- 2026: 63 · 2025: 83 · 2024: 41
- FWCI
- 81.96
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Ulla Tuominen Foundation
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Problem and Project Based Learning
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- prompt engineering, large language models, AI-assisted code generation, programming education, computational thinking, student perceptions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.