Challenges and Limitations of Using Artificial Intelligence Generated Content (AIGC) with ChatGPT in Programming Curriculum: A Systematic Literature Review
Open accessAbstract
This systematic literature review examined the challenges and limitations of integrating Artificial Intelligence Generated Content (AIGC) tools into programming curricula. Following Kitchenham's framework, a comprehensive search was conducted across ACM Digital Library and Web of Science databases to select 22 relevant peer-reviewed papers. Key issues identified included increased plagiarism risk due to AI-generated unique code, accuracy and reliability concerns of AI outputs, privacy and data security risks, curriculum changes to incorporate prompt engineering and critical thinking, and challenges in traditional assessment methods. Additionally, the review highlighted the potential over-reliance on AI tools, which could hinder the development of fundamental programming skills. Implementation challenges and inadequate training for educators further complicate the integration process. Addressing these challenges is crucial for leveraging the benefits of AIGC tools while maintaining academic integrity and enhancing learning outcomes in programming education. Future research should focus on developing strategies to mitigate these issues and ensure the responsible use of AI in education.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The mixed-methods approach was selected because it allows triangulation of findings through complementary data sources, enhancing validity and providing richer insights than either method alone.
“The mixed-methods approach was selected because it allows for triangulation of findings through complementary data sources, enhancing validity and providing richer insights than either method alone2.”
From Integrating AI-generated content tools in higher education: a comparative analysis of interdisciplinary learning outcomes (2025) · cites this paper as [2]
The Methodology section (page unavailable) describes a systematic literature review using Kitchenham’s framework and a search of two publication databases. Searches for “mixed-methods,” “triangulation,” and “qualitative quantitative” found no passage explaining triangulation through complementary data sources or its claimed benefits for validity and insight. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · cited 2× · see citing works
“Similarly, Sarsa et al. [11] found that while OpenAI Codex’s explanations covered most of the code, they contained inaccuracies, particularly in explaining comparison and branching conditionals.”
Lau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241 · cited 4× · see citing works
“Lau and Guo [9] highlighted the need for a substantial overhaul of the types of questions and tasks assigned to students.”
InfluentialEmpowering education development through AIGC: A systematic literature review
Chen, X., Hu, Z., & Wang, C. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12549-7 · Cited by 222 · cited 1× · see citing works
“For instance, Chen et al. [1] conducted a systematic literature review on the macro development status and future trends of AIGC in education, highlighting its transformative potential and identifying primary themes in research.”
Sun, D., Boudouaia, A., Zhu, C., & Li, Y. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00446-5 · Cited by 187 · cited 2× · see citing works
“Similarly, Sun et al. [25] reported inaccurate codes arising from un-updated Python libraries, insufficient code compatibility, and inadequate decomposition of large projects.”
My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E. A., Prather, J., & Becker, B. A. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576134 · Cited by 186 · see citing works
ChatGPT: Challenges and Benefits in Software Programming for Higher Education
da Silva, C. A. G., Ramos, F. N., de Moraes, R. V., & dos Santos, E. L. (2024) · Sustainability · doi:10.3390/su16031245 · Cited by 179 · cited 3× · see citing works
“Similarly, da Silva et al. [22] observed that a notable percentage of students believed that the immediate answers provided by ChatGPT could foster a culture of dependency, replacing the pursuit of understanding with a superficial grasp of concepts.”
InfluentialSavelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141 · cited 2× · see citing works
“Savelka et al. [12] highlighted the genuine concern that learners might rely excessively on ChatGPT models for completing course assignments, thus compromising their ability to develop critical thinking skills.”
The use of Artificial intelligence in school science: a systematic literature review
Heeg, D. M., & Avraamidou, L. (2023) · Educational Media International · doi:10.1080/09523987.2023.2264990 · Cited by 118 · see citing works
Savelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588792 · Cited by 109 · cited 2× · see citing works
“Savelka et al. [13] recommended that instructors consider increasing coding tasks involving nuanced output formatting or practical projects to discourage reliance on AI tools for end-to-end solutions.”
Ouh, E. L., Gan, B. K. S., Shim, K. J., & Wlodkowski, S. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588794 · Cited by 101 · cited 1× · see citing works
“Furthermore, Ouh et al. [10] emphasized that ChatGPT’s text-based input limitations hindered its ability to effectively manage exercises requiring the interpretation of API documentation and UML diagrams.”
Jing, Y., Wang, H., Chen, X., & Wang, C. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-02751-w · Cited by 82 · see citing works
Piccolo, S., Denny, P. C., Luxton-Reilly, A., Payne, S., & Ridge, P. G. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011511 · Cited by 81 · cited 6× · see citing works
“Piccolo et al. [20] emphasized the importance of shifting pedagogical practices towards helping students understand and evaluate the code generated by AIGC tools, ensuring it meets specified requirements, and adapting it to different library versions.”
InfluentialChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course
Popovici, M.-D. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2269006 · Cited by 70 · cited 2× · see citing works
“For instance, Popovici [21] found that although ChatGPT provided correct answers in 68% of the cases, only around half were legible solutions that could benefit students.”
ChatGPT in a programming course: benefits and limitations
Bringula, R. P. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1248705 · Cited by 65 · cited 3× · see citing works
“Bringula [15] observed that teachers relying solely on AI responses might not develop their critical thinking skills, neglecting other educational materials.”
InfluentialInvestigating the Potential of GPT-3 in Providing Feedback for Programming Assessments
Balse, R., Valaboju, B., Singhal, S., Warriem, J. M., & Prasad, P. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588852 · Cited by 63 · see citing works
Potentials of ChatGPT in Computer Programming: Insights from Programming Instructors
Husain, A. (2024) · Journal of Information Technology Education Research · doi:10.28945/5240 · Cited by 57 · cited 5× · see citing works
“Husain [17] highlighted that ChatGPT might generate inaccurate or incorrect code that did not function properly when entered into programming software, which posed a significant challenge for educators who must ensure the reliability of such tools.”
InfluentialGPT-3 vs Object Oriented Programming Assignments: An Experience Report
Cipriano, B. P., & Alves, P. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588814 · Cited by 54 · see citing works
Dunder, N., Lundborg, S., Wong, J., & Viberg, O. (2024) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3636555.3636882 · Cited by 42 · cited 3× · see citing works
“As Dunder et al. [16] highlighted, there was a pressing need for professional development programs that enhanced AI digital competencies among educators.”
InfluentialAI-Enhanced Auto-Correction of Programming Exercises: How Effective is GPT-3.5?
Azaiz, I., Deckarm, O., & Strickroth, S. (2023) · International Journal of Engineering Pedagogy (iJEP) · doi:10.3991/ijep.v13i8.45621 · Cited by 38 · cited 1× · see citing works
“Azaiz et al. [4] and Sarsa et al. [11] noted the necessity of incorporating prompt engineering into the curriculum.”
Detecting AI assisted submissions in introductory programming via code anomaly
Karnalim, O., Toba, H., & Johan, M. C. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12520-6 · Cited by 35 · cited 2× · see citing works
“Similarly, Karnalim et al. [18] discussed the potential dependence on AI tools for task completion, which could detract from the learning experience.”
KOGI: A Seamless Integration of ChatGPT into Jupyter Environments for Programming Education
Kuramitsu, K., Obara, Y., Sato, M., & Obara, M. (2023) · ACM SIGPLAN International Symposium on SPLASH-E (SPLASH-E) · doi:10.1145/3622780.3623648 · Cited by 30 · cited 1× · see citing works
“Kuramitsu et al. [8] expressed concerns about the extensive use of AI, fearing that students might overuse AI tools, thus impairing their learning outcomes.”
Wieser, M., Schoeffmann, K., Stefanics, D., Bollin, A., & Pasterk, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44900-0_4 · Cited by 26 · see citing works
ChatGPT and Python programming homework
Ellis, M. E., Casey, K. M., & Hill, G. E. (2024) · Decision Sciences Journal of Innovative Education · doi:10.1111/dsji.12306 · Cited by 23 · cited 1× · see citing works
“Ellis et al. [23] highlighted that this challenge was worsened by the difficulty distinguishing between student-written and AI-generated code, as the error patterns were not distinct enough to create reliable plagiarism detection tools.”
GPT-Empowered Personalized eLearning System for Programming Languages
Jin, J., & Kim, M. (2023) · Applied Sciences · doi:10.3390/app132312773 · Cited by 21 · cited 2× · see citing works
“For instance, Jin and Kim [19] observed a significant limitation related to the occasional latency in the generation of course content and the grading of student submissions, which ranged from 3 to 8 seconds.”
Flagged first, then by citations
Cited by
Sorted by citations
Yan, Z., & Qianjun, T. (2025) · Scientific Reports · doi:10.1038/s41598-025-10941-y · Cited by 21 · see citing works
“…knowledge (TPACK) H3 Cross-disciplinary variation in learning outcomes follows the function: The mixed-methods approach was selected because it allows for triangulation of findings through complementary data sources, enhancing validity and providing richer insights than either method alone 2 .”
InfluentialChang, C. I., Choi, W. C., & Choi, I. C. (2024) · International Conference on Big Data and Education (ICBDE) · doi:10.1145/3704289.3704301 · Cited by 17 · see citing works
“However, recent studies highlight challenges such as increased plagiarism, over-reliance on AI, and the need for curriculum changes to include prompt engineering and critical thinking [2].”
Self-citationStoyanova, D., Stoyanova-Petrova, S. V., Shotarova, S., Lyubomirov, S. Y., & Mileva, N. S. (2025) · Education Sciences · doi:10.3390/educsci16010019 · Cited by 12 · see citing works
Chang, C. I., Choi, W. C., Choi, I. C., & Lei, H. (2024) · International Conference on Education and E-Learning (ICEEL) · doi:10.1145/3719487.3719519 · Cited by 6 · see citing works
Self-citationChang, C., & Choi, W. C. (2025) · International Conference on Big Data and Education (ICBDE) · doi:10.1145/3786183.3786192 · Cited by 3 · see citing works
Self-citationChoi, W. C., Peng, J., Choi, I. C., Lei, H., Lam, L. C., & Chang, C. I. (2025) · doi:10.1109/icaie64856.2025.11158328 · Cited by 2 · see citing works
Self-citationK Means Clustering for Evaluating Practice Modes in an AI Education Application Course
Li, Y. (2025) · International Conference on Educational Technology and Artificial Intelligence (ICETAI) · doi:10.1145/3766557.3766621
“This mismatch between theory and practice can stunt students’ confidence and curtail their ability to transfer AI techniques across diverse contexts [2].”
Copyright Issues and Innovation Incentives in AI-Generated Content (AIGC)
Zhao, H., & Zhang, Y. (2025) · Innovation Management Practices · doi:10.71204/eg27ey53
The Impact of ChatGPT on Language Assessment in ELT
Tüm, G. (2026) · Sınırsız Eğitim ve Araştırma Dergisi · doi:10.29250/sead.1753516
Choi, W. C., Chang, C. I., & Choi, H. F. (2025) · Artificial Intelligence and Cloud Computing Conference (AICCC) · doi:10.1145/3789982.3790043
“However, ensuring these AI tools promote genuine learning (rather than simply giving students answers) remains challenging [3].”
InfluentialSelf-citationKadams, A. A., Oyelere, S. S., & Omeh, C. B. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811643 · Cited by 1 · see citing works
- Authors
- Chang, C. I., Choi, W. C., & Choi, I. C.
- Affiliations
- Golden Gate University (US)
- Illinois Institute of Technology (US)
- City University of Macau (MO)
- Venue
- Proceedings of the 2024 7th Artificial Intelligence and Cloud Computing Conference
- Publisher
- ACM
- Published
- 2024-12-14
- Pages
- 372-378
- OpenAlex
- W4412119749
- Cited by
- 11
- References
- 24
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 2
- FWCI
- 0.83
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Artificial Intelligence and Cloud Computing Conference (AICCC) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- AI-generated content, ChatGPT, programming education, curriculum integration, plagiarism, AI-generated code, prompt engineering, critical thinking, assessment methods, academic integrity, data privacy, teacher professional development
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
