Analysis of Learning Behaviors and Outcomes for Students with Different Knowledge Levels: A Case Study of Intelligent Tutoring System for Coding and Learning (ITS-CAL)

Abstract

With the rapid development of generative AI technology, programming learning aids have become essential resources for enhancing students’ programming capabilities. This study developed an intelligent tutoring system, ITS-CAL, powered by a large language model (LLM) to provide students with immediate and hierarchical learning feedback, particularly in scenarios with limited class time and large student populations. The system helps students overcome challenges encountered during the learning process. A mixed-method approach, combining quantitative and qualitative analyses, was employed to investigate the usage patterns of the system’s three primary functions—Hint, Debug, and User-defined Question—and their impact on learning outcomes among students with varying knowledge levels. The results indicated that students with high knowledge levels tended to use the Hint and User-defined Question functions moderately, while those with lower knowledge levels heavily relied on the Hint function but did not achieve significant improvements in learning outcomes. Overall, students who used ITS-CAL in moderation achieved the highest pass rate (72.22%), whereas excessive reliance on ITS-CAL appeared to diminish independent problem-solving abilities. Additionally, students generally provided positive feedback on the system’s convenience and its role as a learning aid. However, they highlighted areas for improvement, particularly in the Debug function and the quality of Hint content. This study contributes to the field by demonstrating the application potential of LLMs in programming education and offering valuable empirical insights for designing future programming learning assistance systems.