Recommendations to Create Programming Exercises to Overcome ChatGPT
Open accessAbstract
Large language models, such as ChatGPT, possess the potential to revolutionize educational practices across various domains. Nonetheless, the deployment of these models can inadvertently foster academic dishonesty due to their facile accessibility.In practical courses like programming, where hands-on experience is crucial for learning, relying solely on ChatGPT can hinder students’ ability to engage with the exercises, consequently impeding the attainment of learning outcomes.This paper conducts an experimental analysis of GPT 3.5 and GPT 4, gauging their proficiencies and constraints in resolving a compendium of 22 programming exercises. We discern and categorize exercises based on ChatGPT’s ability to furnish viable solutions, alongside those that remain unaddressed. Moreover, an evaluation of the malleability of the solutions proposed by ChatGPT is undertaken. Subsequently, we propound a series of recommendations aimed at curtailing undue dependence on ChatGPT, thereby fostering authentic competency development in programming. The efficaciousness of these recommendations is underpinned by their integration into the design and delivery of an examination as part of the corresponding course.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552 · see citing works
Self-citationChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · see citing works
Self-citationTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Chatting and cheating: Ensuring academic integrity in the era of ChatGPT
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2190148 · Cited by 3,087 · see citing works
Chatting and Cheating. Ensuring academic integrity in the era of ChatGPT
Cotton, D., Cotton, P., & Shipway, J. R. (2023) · doi:10.35542/osf.io/mrz8h · Cited by 3,079 · see citing works
What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023) · Smart Learning Environments · doi:10.1186/s40561-023-00237-x · Cited by 2,003 · see citing works
Chatting about ChatGPT: how may AI and GPT impact academia and libraries?
Lund, B. D., & Wang, T. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0009 · Cited by 1,355 · see citing works
Chatting about ChatGPT: How May AI and GPT Impact Academia and Libraries?Preprint
Lund, B. D., & Ting, W. (2023) · Social Science Research Network · doi:10.2139/ssrn.4333415 · Cited by 1,089 · see citing works
Learning and Teaching Programming: A Review and Discussion
Robins, A., Rountree, J., & Rountree, N. (2003) · Computer Science Education · doi:10.1076/csed.13.2.137.14200 · Cited by 1,048 · see citing works
Open AI in Education, the Responsible and Ethical Use of ChatGPT Towards Lifelong Learning
Mhlanga, D. (2023) · doi:10.2139/ssrn.4354422 · Cited by 719 · see citing works
Open AI in Education, the Responsible and Ethical Use of ChatGPT Towards Lifelong Learning
Mhlanga, D. (2023) · Sustainable Development Goals Series · doi:10.1007/978-3-031-37776-1_17 · Cited by 718 · see citing works
Using cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559 · see citing works
A Systematic Literature Review on Teaching and Learning Introductory Programming in Higher Education
Medeiros, R. P., Ramalho, G. L., & Falcão, T. P. (2018) · IEEE Transactions on Education · doi:10.1109/te.2018.2864133 · Cited by 389 · see citing works
Use chat gpt to solve programming bugs
Surameery, N. M. S., & Shakor, M. Y. (2023) · International Journal of Information technology and Computer Engineering · doi:10.55529/ijitc.31.17.22 · Cited by 382 · see citing works
An Analysis of the Automatic Bug Fixing Performance of ChatGPT
Sobania, D., Briesch, M., Hanna, C., & Petke, J. (2023) · 2023 IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1109/apr59189.2023.00012 · Cited by 338 · see citing works
Lau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241 · see citing works
A Computer Wrote this Paper: What ChatGPT Means for Education, Research, and WritingPreprint
Shaver, L. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4338981 · Cited by 141 · see citing works
Machine learning based feedback on textual student answers in large courses
Bernius, J. P., Krusche, S., & Bruegge, B. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100081 · Cited by 75 · see citing works
Self-citationUsing cognitive psychology to understand GPT-3Preprint
Binz, M., & Schulz, E. (2022) · doi:10.31234/osf.io/6dfgk · Cited by 62 · see citing works
Finding Plagiarisms among a Set of Programs with JPlag
Malpohl, G., Prechelt, L., & Philippsen, M. (2002) · Verlag der Technischen Universität Graz · doi:10.3217/jucs-008-11-1016 · Cited by 56 · see citing works
GPT-4 vs. GPT-3.5: A Concise Showdown
Koubaa, A. (2023) · doi:10.20944/preprints202303.0422.v1 · Cited by 37 · see citing works
Increasing the Interactivity in Software Engineering MOOCs - A Case Study
Krusche, S., & Seitz, A. (2019) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2019.915 · Cited by 22 · see citing works
Self-citationChaordic Learning: A Case Study
Krusche, S., Bruegge, B., Camilleri, I., Krinkin, K. V., Seitz, A., & Wobker, C. (2017) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1109/icse-seet.2017.21 · Cited by 18 · see citing works
Self-citationA Machine Learning Approach for Suggesting Feedback in Textual Exercises in Large Courses
Bernius, J. P., Krusche, S., & Bruegge, B. (2021) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3430895.3460135 · Cited by 14 · see citing works
Self-citationGPT-4 underlines mismatch on AI policy and innovation
Analytica, O. (2023) · Emerald expert briefings · doi:10.1108/oxan-es276760 · Cited by 12 · see citing works
Towards the Automation of Grading Textual Student Submissions to Open-ended Questions
Bernius, J. P., Kovaleva, A., Krusche, S., & Bruegge, B. (2020) · European Conference on Software Engineering Education (ECSEE) · doi:10.1145/3396802.3396805 · Cited by 5 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 6
- [6]6. S. Biswas, "Role of chatgpt in computer programming.: Chatgpt in computer programming.," Mesopotamian Journal of Computer Science, vol. 2023, pp. 8-16, 2023.
- [11]11. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
- [12]12. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., "Language mod- els are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
- [14]14. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
- [20]20. S. Jalil, S. Rafi, T. D. LaToza, K. Moran, and W. Lam, "Chatgpt and software testing education: Promises & perils," in International Conference on Software Testing, Verification and Validation Workshops, pp. 4130-4137, IEEE, 2023.
- [23]23. S. Krusche and A. Seitz, "ArTEMiS: An Automatic Assessment Man- agement System for Interactive Learning," in 49th Technical Symposium on Computer Science Education, pp. 284-289, ACM, 2018.
- Authors
- Berrezueta-Guzman, J., & Krusche, S.
- Affiliations
- Technical University of Munich (DE)
- Venue
- IEEE International Conference on Software Engineering Education and Training (CSEE&T)
- Publisher
- Center for Open Science
- Published
- 2023-08-24
- Pages
- 147-151
- ISSN
- 1093-0175 (ISSN-L)
- OpenAlex
- W4386323456
- Cited by
- 27
- References
- 31
- Citations by year
- 2026: 4 · 2025: 9 · 2024: 12 · 2023: 2
- FWCI
- 8.13
- Type
- conference paper
- Language
- en
- Open access
- green
- doi.org
- Locations
- IEEE International Conference on Software Engineering Education and Training (CSEE&T) (publishedVersion)
- EdArXiv (OSF Preprints) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Social Sciences › Safety Research › Academic integrity and plagiarism
- Keywords
- ChatGPT, large language models, programming education, programming exercises, academic dishonesty, GPT-3.5, GPT-4, exercise design, student competence development, programming assessment, generative AI in education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
