The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Prompt Problems: A New Programming Exercise for the Generative AI Era
Open accessAbstract
Large language models (LLMs) are revolutionizing the field of computing education with their powerful code-generating capabilities. Traditional pedagogical practices have focused on code writing tasks, but there is now a shift in importance towards reading, comprehending and evaluating LLM-generated code. Alongside this shift, an important new skill is emerging -- the ability to solve programming tasks by constructing good prompts for code-generating models. In this work we introduce a new type of programming exercise to hone this nascent skill: 'Prompt Problems'. Prompt Problems are designed to help students learn how to write effective prompts for AI code generators. A student solves a Prompt Problem by crafting a natural language prompt which, when provided as input to an LLM, outputs code that successfully solves a specified programming task. We also present a new web-based tool called Promptly which hosts a repository of Prompt Problems and supports the automated evaluation of prompt-generated code. We deploy Promptly in one CS1 and one CS2 course and describe our experiences, which include student perceptions of this new type of activity and their interactions with the tool. We find that students are enthusiastic about Prompt Problems, and appreciate how the problems engage their computational thinking skills and expose them to new programming constructs. We discuss ideas for the future development of new variations of Prompt Problems, and the need to carefully study their integration into classroom practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Further, novice learners of programming may become overwhelmed and confused by generated code, which could add to the high levels of frustration that are common in introductory programming courses [71,188].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [71]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although LLMs can automate routine programming tasks, their limitations become apparent when addressing nuanced or ill-structured problems (Zhang et al., 2023;Denny et al. 2024).”
From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Denny et al. 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Explicit instruction on how to critically assess, refine, and debug AI-generated code fosters the development of essential competencies -ranging from algorithmic reasoning to software testing (Denny et al., 2024;Kendon et al., 2023).”
From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Denny et al., 2024
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Although generative systems are stateless in that they do not retain context across interactions unless explicitly prompted [107,108]”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [107]
Experimental — accuracy not yet measuredIn the citing paper
“Indeed, some recent papers found in the second phase snowballing focused on introducing tools around LLMs, such as CodeHelp [127] and Promptly [71].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [71]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838 · cited 1×
“White et al. present a prompt structuring framework for constructing prompts that can be applied across problem domains, and demonstrate how prompts can be constructed from patterns [30].”
BackgroundAutomatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
Self-citationThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463
Self-citationBecker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401 · cited 1×
“In addition to the opportunities they present, educators have voiced concerns around the potential misuse of these models for plagiarism, and over-reliance on AI-generated code by beginners [3], leading to a possible erosion of traditional coding skills [9].”
BackgroundSelf-citationThe Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · cited 1×
“The advent of large language models (LLMs) that can generate code is having a rapid and significant impact on computing education practice, particularly at the introductory level [24].”
BackgroundSelf-citationStudying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287
Self-citationMalinka, K., Peresíni, M., Firc, A., Hujnák, O., & Janus, F. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588827 · Cited by 287
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
Self-citationLau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241
Research works printed but not linked: 2
- Leo Porter and Daniel Zingaro. 2023. Learn AI-Assisted Python Programming: With Github Copilot and ChatGPT. Manning, Shelter Island, NY.
- 10.1007/978--3-031--11647--6_127 · doi:10.1007/978--3-031--11647--6_127
Cited by
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
Self-citationBengesi, S., El-Sayed, H., Sarker, M. K., Houkpati, Y., Irungu, J., & Oladunni, T. (2024) · IEEE Access · doi:10.1109/access.2024.3397775 · Cited by 362
“This prompt is crucial in determining the generated output, making prompt engineering a vital skill [142].”
BackgroundComputing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“Depending on the prompt, the resulting programs used varied programmatic structures, while ultimately favoring expected methods for each problem variation (i.e., for-loops for processing lists, and while-loops for processing standard input).”
BackgroundSelf-citationAI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · arXiv · doi:10.48550/arxiv.2505.10468 · Cited by 295
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 671
“• Prompt Dependency and Statelessness: Although generative systems are stateless in that they do not retain context across interactions unless explicitly prompted [107,108], recent advancements like GPT-4.”
Metadata
- Authors
- Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N.
- Affiliations
- University of Auckland (NZ)
- Abilene Christian University (US)
- University of Toronto (CA)
- University College Dublin (IE)
- ORCID
- Venue
- Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2024-03-07
- Pages
- 296-302
- OpenAlex
- W4392564669
- Cited by
- 187
- References
- 27
- Citations by year
- 2026: 63 · 2025: 83 · 2024: 41
- FWCI
- 81.96
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Ulla Tuominen Foundation
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Problem and Project Based Learning
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- prompt engineering, large language models, AI-assisted code generation, programming education, computational thinking, student perceptions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Prompt Problems: A New Programming Exercise for the Generative AI Era. https://app.cochat.ai/papers/10.1145/3626252.3630909
- The paper
- Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024). Prompt Problems: A New Programming Exercise for the Generative AI Era. Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1. https://doi.org/10.1145/3626252.3630909
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.