CoChat
Sign Up
Cited by 27Teaching and Learning ProgrammingArtificial Intelligence in EducationAcademic integrity and plagiarism

Recommendations to Create Programming Exercises to Overcome ChatGPT

Open access
Berrezueta-Guzman, Jonnathan, Krusche, Stephan·2023·2023 IEEE 35th International Conference on Software Engineering Education and Training (CSEE&T)
DOI 10.1109/cseet58097.2023.00031
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models, such as ChatGPT, possess the potential to revolutionize educational practices across various domains. Nonetheless, the deployment of these models can inadvertently foster academic dishonesty due to their facile accessibility.In practical courses like programming, where hands-on experience is crucial for learning, relying solely on ChatGPT can hinder students’ ability to engage with the exercises, consequently impeding the attainment of learning outcomes.This paper conducts an experimental analysis of GPT 3.5 and GPT 4, gauging their proficiencies and constraints in resolving a compendium of 22 programming exercises. We discern and categorize exercises based on ChatGPT’s ability to furnish viable solutions, alongside those that remain unaddressed. Moreover, an evaluation of the malleability of the solutions proposed by ChatGPT is undertaken. Subsequently, we propound a series of recommendations aimed at curtailing undue dependence on ChatGPT, thereby fostering authentic competency development in programming. The efficaciousness of these recommendations is underpinned by their integration into the design and delivery of an examination as part of the corresponding course.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cseet58097.2023.00031

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • When producing AI-generated answers without engaging in analysis or reflection, students miss opportunities to develop their skills and knowledge, and to receive meaningful feedback on their progress.

    “When producing AI-generated answers without engaging in analysis or reflection, students miss opportunities to develop their skills and knowledge, and to receive meaningful feedback on their progress [40, 224].”

    From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [40]

    The Conclusion and Future Work section says that students who solve exercises without analyzing and reflecting may face long-term problems, and the Findings section emphasizes that programming skills require practice. The paper does not say that using AI-generated answers deprives students of meaningful feedback on their progress; its discussion of continuous feedback concerns hidden test cases. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Several papers described approaches that educators have made to safeguard academic integrity, such as weighting invigilated assessment more heavily, banning GenAI use in specific contexts, and explicitly demonstrating to students both the capabilities and limitations of these tools.

    “Several papers described approaches that educators have made to safeguard academic integrity, such as weighting invigilated assessment more heavily, banning GenAI use in specific contexts, and explicitly demonstrating to students both the capabilities and limitations of these tools [40, 198].”

    From Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education (2025) · cites this paper as [40]

    The section on preparing the examination describes an onsite, supervised exam, and the Conclusion says the authors' exercise-design recommendations aim to protect academic integrity. The Conclusion also notes that some universities have banned ChatGPT. Our copy does not say that educators weighted invigilated assessments more heavily or explicitly demonstrated the tools’ capabilities and limitations to students. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
  2. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., "Language mod- els are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
  3. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
  4. O. Analytica, "Gpt-4 underlines mismatch on ai policy and innovation," Emerald Expert Briefings, 2023.
  5. R. Sutton and A. Barto, Reinforcement learning: An introduction. MIT press, 2018.
  6. L. Prechelt, G. Malpohl, M. Philippsen, et al., "Finding plagiarisms among a set of programs with jplag.," vol. 8, no. 11, p. 1016, 2002.

Cited by

Sorted by citations

Authors
Berrezueta-Guzman, J., & Krusche, S.
Affiliations
  • Technical University of Munich
  • Technical University of Munich (DE)
ORCID
Krusche, Stephan
Venue
2023 IEEE 35th International Conference on Software Engineering Education and Training (CSEE&T)
Publisher
IEEE
Published
2023-08
Pages
147-151
ISSN
1093-0175 (ISSN-L)
DOI
10.1109/cseet58097.2023.00031
OpenAlex
W4386323456
Cited by
27
References
26
Citations by year
2026: 4 · 2025: 9 · 2024: 12 · 2023: 2
FWCI
8.16
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE International Conference on Software Engineering Education and Training (CSEE&T) (publishedVersion)
  • EdArXiv (OSF Preprints) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Social Sciences › Safety Research › Academic integrity and plagiarism
Keywords
ChatGPT, large language models, programming education, programming exercises, academic dishonesty, GPT-3.5, GPT-4, exercise design, student competence development, programming assessment, generative AI in education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review