Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.

Prompt Problems: A New Programming Exercise for the Generative AI Era

Open access
Denny, Paul, Leinonen, Juho, Prather, James, Luxton-Reilly, Andrew, Amarouche, Thezyrie, Becker, Brett A., Reeves, Brent N. · 2024 · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1

Abstract

Large language models (LLMs) are revolutionizing the field of computing education with their powerful code-generating capabilities. Traditional pedagogical practices have focused on code writing tasks, but there is now a shift in importance towards reading, comprehending and evaluating LLM-generated code. Alongside this shift, an important new skill is emerging -- the ability to solve programming tasks by constructing good prompts for code-generating models. In this work we introduce a new type of programming exercise to hone this nascent skill: 'Prompt Problems'. Prompt Problems are designed to help students learn how to write effective prompts for AI code generators. A student solves a Prompt Problem by crafting a natural language prompt which, when provided as input to an LLM, outputs code that successfully solves a specified programming task. We also present a new web-based tool called Promptly which hosts a repository of Prompt Problems and supports the automated evaluation of prompt-generated code. We deploy Promptly in one CS1 and one CS2 course and describe our experiences, which include student perceptions of this new type of activity and their interactions with the tool. We find that students are enthusiastic about Prompt Problems, and appreciate how the problems engage their computational thinking skills and expose them to new programming constructs. We discuss ideas for the future development of new variations of Prompt Problems, and the need to carefully study their integration into classroom practice.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3626252.3630909

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 An example Prompt Problem that displays the data visually to prevent copying and pasting of the description into an LLM. The goal is to swap the top-left and bottomright non-overlapping quadrants of the matrix.
Figure 1 An example Prompt Problem that displays the data visually to prevent copying and pasting of the description into an LLM. The goal is to swap the top-left and bottomright non-overlapping quadrants of the matrix.CC-BY · From the PDF
Figure 2 Interface layout for a Prompt Problem within the web-based Promptly tool (with figure annotations added in blue). The layout is compressed for space reasons.
Figure 2 Interface layout for a Prompt Problem within the web-based Promptly tool (with figure annotations added in blue). The layout is compressed for space reasons.CC-BY · From the PDF
Figure 3 Each line represents all submissions made by a student for the CS1-1 problem. Blue dots denote every successful submission; an orange X denotes final unsuccessful submission. Several students submit more than one successful prompt, indicating experimentation with the problem.
Figure 3 Each line represents all submissions made by a student for the CS1-1 problem. Blue dots denote every successful submission; an orange X denotes final unsuccessful submission. Several students submit more than one successful prompt, indicating experimentation with the problem.CC-BY · From the PDF
Figure 4 Each line represents all submissions made by a student for the CS2-1 problem.
Figure 4 Each line represents all submissions made by a student for the CS2-1 problem.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 27 references

Research works printed but not linked: 2

  1. Leo Porter and Daniel Zingaro. 2023. Learn AI-Assisted Python Programming: With Github Copilot and ChatGPT. Manning, Shelter Island, NY.
  2. 10.1007/978--3-031--11647--6_127 · doi:10.1007/978--3-031--11647--6_127

Cited by

See all 194

Metadata

Authors
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N.
Affiliations
  • University of Auckland (NZ)
  • Abilene Christian University (US)
  • University of Toronto (CA)
  • University College Dublin (IE)
Venue
Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1
Publisher
ACM
Published
2024-03-07
Pages
296-302
OpenAlex
W4392564669
Cited by
187
References
27
Citations by year
2026: 63 · 2025: 83 · 2024: 41
FWCI
81.96
Type
conference paper
Language
en
Open access
Locations
  • ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Ulla Tuominen Foundation
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Social Sciences › Education › Problem and Project Based Learning
  • Social Sciences › Education › Artificial Intelligence in Education
Keywords
prompt engineering, large language models, AI-assisted code generation, programming education, computational thinking, student perceptions

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Prompt Problems: A New Programming Exercise for the Generative AI Era. https://app.cochat.ai/papers/10.1145/3626252.3630909
The paper
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024). Prompt Problems: A New Programming Exercise for the Generative AI Era. Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1. https://doi.org/10.1145/3626252.3630909

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.