CoChat

The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.

CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes

Open access
Liffiton, Mark, Sheese, Brad E, Savelka, Jaromir, Denny, Paul · 2023 · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research

Abstract

Computing educators face significant challenges in providing timely support to students, especially in large class settings. Large language models (LLMs) have emerged recently and show great promise for providing on-demand help at a large scale, but there are concerns that students may over-rely on the outputs produced by these models. In this paper, we introduce CodeHelp, a novel LLM-powered tool designed with guardrails to provide on-demand assistance to programming students without directly revealing solutions. We detail the design of the tool, which incorporates a number of useful features for instructors, and elaborate on the pipeline of prompting strategies we use to ensure generated outputs are suitable for students. To evaluate CodeHelp, we deployed it in a first-year computer and data science course with 52 students and collected student interactions over a 12-week period. We examine students’ usage patterns and perceptions of the tool, and we report reflections from the course instructor and a series of recommendations for classroom use. Our findings suggest that CodeHelp is well-received by students who especially value its availability and help with resolving errors, and that for instructors it is easy to deploy and complements, rather than replaces, the support that they provide to students.

Overview

Integrity signals

Registry

Crossref · 10.1145/3631802.3631830

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 34 references

Research works printed but not linked: 10

  1. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, 2021. Evaluating large language models trained on code. arxiv:2107.03374 [cs.LG] · arXiv
  2. Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves. 2023. Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators. arxiv:2307.16364 [cs.HC] · arXiv
  3. Paul Denny James Prather Brett A. Becker James Finnie-Ansley Arto Hellas Juho Leinonen Andrew Luxton-Reilly Brent N. Reeves Eddie Antonio Santos and Sami Sarsa. 2023. Computing Education in the Era of Generative AI. arxiv:2306.02608 [cs.CY] · arXiv
  4. Charles Koutcheme, Sami Sarsa, Juho Leinonen, Arto Hellas, and Paul Denny. 2023. Automated Program Repair Using Generative Models for Code Infilling. In Artificial Intelligence in Education, Ning Wang, Genaro Rebolledo-Mendez, Noboru Matsuda, Olga C. Santos, and Vania Dimitrova (Eds.). Springer Nature Switzerland, Cham, 798–803.
  5. Chinedu Wilfred Okonkwo and Abejide Ade-Ibijola. 2021. Python-Bot: A Chatbot for Teaching Python Programming. Engineering Letters 29 (02 2021), 25–34.
  6. Chinedu Wilfred Okonkwo and Abejide Ade-Ibijola. 2022. Revision-Bot: A Chatbot for Studying Past Questions in Introductory Programming. IAENG International Journal of Computer Science 49, 3 (2022).
  7. Zachary A. Pardos and Shreya Bhandari. 2023. Learning gain differences between ChatGPT and human tutor generated algebra hints. arxiv:2302.06871 [cs.CY] · arXiv
  8. Margot Rutgers. 2021. Duckbot: A chatbot to assist students in programming tutorials. Master’s thesis. University of Twente.
  9. James Walden, Nicholas Caporusso, and Ludiana Atnafu. 2022. A Chatbot for Teaching Secure Programming. In Proceedings of the EDSIG Conference ISSN, Vol. 2473. 4901.
  10. Terry Yue Zhuo Yujin Huang Chunyang Chen and Zhenchang Xing. 2023. Red teaming ChatGPT via Jailbreaking: Bias Robustness Reliability and Toxicity. arxiv:2301.12867 [cs.CL] · arXiv

Cited by

See all 378

Metadata

Authors
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P.
Affiliations
  • Illinois Wesleyan University (US)
  • Carnegie Mellon University, USA
  • University of Auckland (NZ)
Venue
Proceedings of the 23rd Koli Calling International Conference on Computing Education Research
Publisher
ACM
Published
2023-11-13
Pages
1-11
OpenAlex
W4391584331
Cited by
192
References
25
Citations by year
2026: 54 · 2025: 77 · 2024: 59 · 2023: 2
FWCI
57.99
Type
conference paper
Language
en
Open access
Locations
  • Koli Calling International Conference on Computing Education Research (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Social Sciences › Education › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
large language models, guardrails, programming education, prompting strategies, data science education, student perceptions, instructor support, error resolution, overreliance on AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes. https://app.cochat.ai/papers/10.1145/3631802.3631830
The paper
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023). CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes. Proceedings of the 23rd Koli Calling International Conference on Computing Education Research. https://doi.org/10.1145/3631802.3631830

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.