CoChat
Sign Up
Cited by 107

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.

Computer Science Education in ChatGPT Era: Experiences from an Experiment in a Programming Course for Novice Programmers

Kosar, Tomaž, Ostojić, Dragana, Liu, Yu David, Mernik, Marjan·2024·Mathematics
DOI 10.3390/math12050629
Open this paper in CoChat — free account

Abstract

The use of large language models with chatbots like ChatGPT has become increasingly popular among students, especially in Computer Science education. However, significant debates exist in the education community on the role of ChatGPT in learning. Therefore, it is critical to understand the potential impact of ChatGPT on the learning, engagement, and overall success of students in classrooms. In this empirical study, we report on a controlled experiment with 182 participants in a first-year undergraduate course on object-oriented programming. Our differential study divided students into two groups, one using ChatGPT and the other not using it for practical programming assignments. The study results showed that the students’ performance is not influenced by ChatGPT usage (no statistical significance between groups with a p-value of 0.730), nor are the grading results of practical assignments (p-value 0.760) and midterm exams (p-value 0.856). Our findings from the controlled experiment suggest that it is safe for novice programmers to use ChatGPT if specific measures and adjustments are adopted in the education process.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/math12050629

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Example of practical assignment (mandatory in week 5).
Figure 1 Example of practical assignment (mandatory in week 5).CC-BY · From the PDF
Figure 3 Number of participants in Group II that used ChatGPT regularly for practical assignments.
Figure 3 Number of participants in Group II that used ChatGPT regularly for practical assignments.CC-BY · From the PDF
Figure 4 Number of participants in Group I that used ChatGPT regularly for practical assignments.
Figure 4 Number of participants in Group I that used ChatGPT regularly for practical assignments.CC-BY · From the PDF
Figure 5 The positive impact of ChatGPT on course grade (Group I).
Figure 5 The positive impact of ChatGPT on course grade (Group I).CC-BY · From the PDF
Image 5
CC-BY · From the PDF
Figure 7 The understandability of code received from ChatGPT (Group I).
Figure 7 The understandability of code received from ChatGPT (Group I).CC-BY · From the PDF
Image 7
CC-BY · From the PDF
Figure 9 Future use of ChatGPT for programming.
Figure 9 Future use of ChatGPT for programming.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 11 references

Research works printed but not linked: 35

  1. 2023, What ChatGPT and generative AI mean for science, Nature, 614, 214, 10.1038/d41586-023-00340-6 · DOI · Paper page
  2. MacNeil, S., Tran, A., Mogil, D., Bernstein, S., Ross, E., and Huang, Z. (2022, January 7–11). Generating diverse code explanations using the GPT-3 large language model. Proceedings of the 2022 ACM Conference on International Computing Education Research, Virtual. · DOI · Paper page
  3. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.d.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., and Brockman, G. (2021). Evaluating large language models trained on code. arXiv.
  4. Tian, H., Lu, W., Li, T.O., Tang, X., Cheung, S.C., Klein, J., and Bissyandé, T.F. (2023). Is ChatGPT the Ultimate Programming Assistant–How far is it?. arXiv.
  5. Gayoso, 2023, Examining the Impacts of ChatGPT on Student Motivation and Engagement, Soc. Space, 23, 1
  6. Qureshi, B. (2023). Exploring the use of ChatGPT as a tool for learning and assessment in undergraduate computer science curriculum: Opportunities and challenges. arXiv.
  7. Milano, 2023, Large language models challenge the future of higher education, Nat. Mach. Intell., 5, 333, 10.1038/s42256-023-00644-2 · DOI
  8. DeFranco, 2023, Are We Writing for Bots or Humans?, Computer, 56, 13
  9. Cao, J., Li, M., Wen, M., and Cheung, S.C. (2023). A study on prompt design, advantages and limitations of ChatGPT for deep learning program repair. arXiv.
  10. Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., and Yang, D. (2023). Is ChatGPT a general-purpose natural language processing task solver?. arXiv. · DOI
  11. Winslow, 1996, Programming pedagogy—A psychological view, ACM SIGCSE Bull., 28, 17, 10.1145/234867.234872 · DOI
  12. Sánchez-Ruiz, L.M., Moll-López, S., Nuñez-Pérez, A., Moraño-Fernández, J.A., and Vega-Fleitas, E. (2023). ChatGPT Challenges Blended Learning Methodologies in Engineering Education: A Case Study in Mathematics. Appl. Sci., 13. · DOI
  13. Susnjak, T. (2022). ChatGPT: The end of online exam integrity?. arXiv.
  14. Geng, C., Yihan, Z., Pientka, B., and Si, X. (2023). Can ChatGPT Pass An Introductory Level Functional Language Programming Course?. arXiv.
  15. Shoufan, 2023, Can Students without Prior Knowledge Use ChatGPT to Answer Test Questions? An Empirical Study, ACM Trans. Comput. Educ., 23, 45, 10.1145/3628162 · DOI · Paper page
  16. King, 2023, A conversation on artificial intelligence, chatbots, and plagiarism in higher education, Cell. Mol. Bioeng., 16, 1, 10.1007/s12195-022-00754-8 · DOI · Paper page
  17. Chowdhary, K.R. (2020). Fundamentals of Artificial Intelligence, Springer. · DOI · Paper page
  18. King, 2023, The future of AI in medicine: A perspective from a Chatbot, Ann. Biomed. Eng., 51, 291, 10.1007/s10439-022-03121-w · DOI · Paper page
  19. Floridi, 2020, GPT-3: Its nature, scope, limits, and consequences, Minds Mach., 30, 681, 10.1007/s11023-020-09548-1 · DOI · Paper page
  20. Vaswani, 2017, Attention is all you need, Adv. Neural Inf. Process. Syst., 30, 5998
  21. Jeon, 2023, Beyond ChatGPT: A conceptual framework and systematic review of speech-recognition chatbots for language learning, Comput. Educ., 206, 104898, 10.1016/j.compedu.2023.104898 · DOI
  22. White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., and Schmidt, D.C. (2023). A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv.
  23. White, J., Hays, S., Fu, Q., Spencer-Smith, J., and Schmidt, D.C. (2023). ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design. arXiv.
  24. Cabot, 2023, A domain-specific language for describing machine learning datasets, J. Comput. Lang., 76, 101209, 10.1016/j.cola.2023.101209 · DOI
  25. Zorrilla, 2020, Lavoisier: A DSL for increasing the level of abstraction of data selection and formatting in data mining, J. Comput. Lang., 60, 100987, 10.1016/j.cola.2020.100987 · DOI
  26. Kosar, 2018, Program comprehension of domain-specific and general-purpose languages: Replication of a family of experiments using integrated development environments, Empir. Softw. Eng., 23, 2734, 10.1007/s10664-017-9593-2 · DOI
  27. Sonnleitner, 2012, The Genetics Lab. Acceptance and psychometric characteristics of a computer-based microworld to assess complex problem solving, Psychol. Test Assess. Model., 54, 54
  28. Ouh, E.L., Gan, B.K.S., Shim, K.J., and Wlodkowski, S. (2023). ChatGPT, Can You Generate Solutions for my Coding Exercises? An Evaluation on its Effectiveness in an undergraduate Java Programming Course. arXiv. · DOI
  29. Majdinasab, 2023, GitHub Copilot AI pair programmer: Asset or Liability?, J. Syst. Softw., 203, 111734, 10.1016/j.jss.2023.111734 · DOI · Paper page
  30. Asare, 2023, Is GitHub’s Copilot as bad as humans at introducing vulnerabilities in code?, Empir. Softw. Eng., 28, 129, 10.1007/s10664-023-10380-1 · DOI · Paper page
  31. Likert, 1932, A technique for the measurement of attitudes, Arch. Psychol., 22, 55
  32. Feldt, R., and Magazinius, A. (2010, January 1–3). Validity Threats in Empirical Software Engineering Research—An Initial Survey. Proceedings of the 22nd International Conference on Software Engineering & Knowledge Engineering (SEKE’2010), Redwood City, CA, USA.
  33. Bergersen, 2023, Construct Validity in Software Engineering, IEEE Trans. Softw. Eng., 49, 1374, 10.1109/TSE.2022.3176725 · DOI
  34. Shull, 2008, The role of replications in empirical software engineering, Empir. Softw. Eng., 13, 211, 10.1007/s10664-008-9060-1 · DOI
  35. Carver, J.C. (2010, January 2–8). Towards reporting guidelines for experimental replications: A proposal. Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering, Cape Town, South Africa.

Cited by

See all 107

Metadata

Authors
Kosar, T., Ostojić, D., Liu, Y. D., & Mernik, M.
Affiliations
  • Faculty of Electrical Engineering and Computer Science, University of Maribor, Koroška cesta 46, 2000 Maribor, Slovenia
  • Department of Computer Science, State University of New York at Binghamton (SUNY), 4400 Vestal Parkway East, Binghamton, NY 13902, USA
ORCID
Venue
Mathematics
Publisher
MDPI AG
Published
2024-02-21
Volume
12
Issue
5
Pages
629
ISSN
2227-7390
DOI
10.3390/math12050629
Cited by
107
References
11
Language
en
Licence
creativecommons.org/licenses/by/4.0/ (vor)
Funders
  • Slovenian Research Agency (P2-0041)
  • Fulbright Scholar Program (P2-0041)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Computer Science Education in ChatGPT Era: Experiences from an Experiment in a Programming Course for Novice Programmers. https://app.cochat.ai/papers/10.3390/math12050629
The paper
Kosar, T., Ostojić, D., Liu, Y. D., & Mernik, M. (2024). Computer Science Education in ChatGPT Era: Experiences from an Experiment in a Programming Course for Novice Programmers. Mathematics. https://doi.org/10.3390/math12050629