The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index; 1 of the 13 sources its claims cite were checked for support.
Knowledge Tracing in Programming Education Integrating Students’ Questions
Open accessAbstract
Knowledge tracing (KT) in programming education presents unique challenges due to the complexity of coding tasks and the diverse methods students use to solve problems.Although students' questions often contain valuable signals about their understanding and misconceptions, traditional KT models often neglect to incorporate these questions as inputs to address these challenges.This paper introduces SQKT (Students' Question-based Knowledge Tracing), a knowledge tracing model that leverages students' questions and automatically extracted skill information to enhance the accuracy of predicting students' performance on subsequent problems in programming education.Our method creates semantically rich embeddings that capture not only the surfacelevel content of the questions but also the student's mastery level and conceptual understanding.Experimental results demonstrate SQKT's superior performance in predicting student completion across various Python programming courses of differing difficulty levels.In in-domain experiments, SQKT achieved a 33.1% absolute improvement in AUC compared to baseline models.The model also exhibited robust generalization capabilities in cross-domain settings, effectively addressing data scarcity issues in advanced programming courses.SQKT can be used to tailor educational content to individual learning needs and design adaptive learning systems in computer science education.Our code is available at
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
SQKT is a knowledge tracing model for programming education that integrates students’ questions and automatically extracted skill information to predict success on subsequent Python problems. It reports superior in-domain and cross-domain performance over baselines, with up to a 33.1% absolute AUC improvement in in-domain experiments.
Key findings
- In-domain, SQKT achieved AUC 87.1–93.4 across three courses, an absolute improvement of 12.6–20.8 over the best baseline (KTMFF+); the abstract reports up to a 33.1% absolute AUC improvement over baseline models.
- Ablation on Python Basic: full SQKT reached AUC 93.4, ACC 89.2, F1 88.4; removing both question and skill embeddings dropped performance to AUC 85.4, ACC 80.7, F1 82.7, and using only questions or only skills was also suboptimal.
- Cross-domain content-structure generalization improved by an absolute 45.3% AUC over the no-question variant; in data-scarce generalization on Algorithm, the full model improved 11.4% over the in-domain model and 16.7% over the cross-domain no-question model.
- The skill extractor on 100 Python Basic questions achieved precision 0.85, recall 0.88, and F1 0.86, with human annotation agreement at Cohen’s kappa 0.98.
Methods
- Data came from a Korean online programming education platform (January 2022–April 2024), covering four Python courses: Python Basic, First Python, Algorithm, and Python Introduction. Data were split by students into training, validation, and test sets at an 8:1:1 ratio.
- SQKT encodes problem descriptions with BERT-base, code submissions with CodeBERT, student questions with CodeT5 fine-tuned via an educator-response auxiliary loss, and skills with BERT-base. Skills are extracted by a GPT-4o-generated rule-based skill extractor over 36 Python concepts and 19 error types.
- The model uses a fusion layer with triplet loss, six self-attention layers, max-pooling, and a classification head, trained with binary cross-entropy plus question and triplet losses. Baselines include KTMFF, OKT, and their question-augmented variants; evaluation uses AUC, accuracy, and F1, with success defined as score 100 within the course-average number of submissions.
Limitations
- The paper states that no preprocessing was applied to input questions, so inputs are often noisy; and the skill extractor is rule-based rather than statistical machine learning, chosen for interpretability but potentially limiting scalability and adaptation to unseen patterns.
- Text-evident: the skill extractor is not perfectly accurate (precision 0.85, recall 0.88, F1 0.86), and error analysis attributes mispredictions to complexity (55.6%), confusion (40.7%), incompleteness (29.6%), and ambiguity (22.2%). The Algorithm course used for data-scarce generalization is also very small (300 test samples), which may constrain generalization claims.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 of 13 cited sources checked · 1 supported · 0 flagged
- Findingown data§ 5.3 Error Analysis · p. 8
In the error analysis, 'Complexity' was the most prevalent error type, accounting for 55.6% of sampled mispredictions.
“Our analysis shows that 'Complexity' is the most prevalent issue (55.6%)”
Quote in the paperSubstantial claim - Findingown data§ 5.1 In-Domain Results · p. 7
Removing the question loss causes slight drops in performance across the three courses.
“The question loss, derived from the task of predicting educator responses to student questions, impacts performance across the three courses, with slight drops in performance when removed.”
Quote in the paperSubstantial claim - Findingown data§ 5.1 In-Domain Results · p. 7
The two auxiliary losses (question loss and triplet loss) improve performance across diverse programming courses.
“The results in Table 4 validate the importance of these additional objectives in improving performance across diverse programming courses.”
Quote in the paperSubstantial claim - Findingown data§ 5.1 In-Domain Results · p. 7
Student questions provide valuable insights into student performance.
“The improvement of KTMFF+ over KTMFF and OKT+ over OKT reinforces our research motivation that student questions provide valuable insights into student performance.”
Quote in the paperSubstantial claim - Findingown data§ 5.1 In-Domain Results · p. 7
Relying solely on skills or solely on questions performs worse than SQKT.
“Relying solely on either skills (row 3) or questions (row 4) is suboptimal, demonstrating their synergistic contribution.”
Quote in the paperSubstantial claim
- Findingown data§ 5.3 Error Analysis · p. 8
'Confusion' was the second most common error type, accounting for 40.7% of sampled mispredictions.
“'Confusion' is the second most common error type (40.7%)”
Quote in the paperNot a substantial claim - Findingnone given§ 3.1 Multi-feature Inputs · p. 4
Extracted skills substantially improve SQKT's predictive performance.
“We considered this level of accuracy acceptable, as extracted skills substantially improve SQKT's predictive performance (as discussed in the experiment section).”
Quote in the paperNot a substantial claim - Findingown data§ 5.3 Error Analysis · p. 8
'Incompleteness' accounted for 29.6% of sampled mispredictions.
“'Incompleteness' (29.6%)”
Quote in the paperNot a substantial claim - Findingown data§ 1 Introduction · p. 2
Combining auto-extracted Python skills with student questions significantly improves model performance compared to relying solely on natural language questions.
“Our method for combining auto-extracted Python skills with student questions significantly improves model performance compared to relying solely on natural language questions.”
Quote in the paperSubstantial claim - Findingown data§ 5.1 In-Domain Results · p. 7
Removing both questions and skills significantly degrades the model's performance.
“Removing both questions and skills (row 5) significantly degrades the model's performance.”
Quote in the paperSubstantial claim - Findingown data§ 3.1 Multi-feature Inputs · p. 4
A rule-based method is preferable to using GPT on the fly, due to high precision and consistency in skill identification.
“We found that a rulebased method is preferable to using GPT on the fly, due to high precision and consistency in skill identification.”
Quote in the paperSubstantial claim - Findingown data§ Abstract · p. 1
SQKT has superior performance in predicting student completion across various Python programming courses of differing difficulty levels.
“Experimental results demonstrate SQKT's superior performance in predicting student completion across various Python programming courses of differing difficulty levels.”
Quote in the paperSubstantial claim - Findingown data§ 5.3 Error Analysis · p. 8
'Ambiguity' accounted for 22.2% of sampled mispredictions.
“'Ambiguity' (22.2%)”
Quote in the paperNot a substantial claim - Findingown data§ 5.1 In-Domain Results · p. 7
Across the three courses, SQKT consistently outperformed all baselines.
“Across the three courses, SQKT consistently outperformed all baselines.”
Quote in the paperSubstantial claim - Findingown data§ Abstract · p. 1
In in-domain experiments, SQKT achieved a 33.1% absolute improvement in AUC compared to baseline models.
“In indomain experiments, SQKT achieved a 33.1% absolute improvement in AUC compared to baseline models.”
- Also stated in: § 1 Introduction · p. 2 · boosted
“Experimental results show significant improvements over existing methods, with up to a 33.1% absolute improvement in AUC compared to baseline models in in-domain experiments.”
Quote in the paperSubstantial claim - Also stated in: § 1 Introduction · p. 2 · boosted
- Findingreasoning§ 1 Introduction · p. 1
Merely using a transformer to encode students' questions is suboptimal.
“As we will show, merely using a transformer to encode students' questions is suboptimal, because it might not fully represent the patterns of confusion and educational context that could be captured through the interaction between student and educator.”
Quote in the paperNot a substantial claim - Findingown data§ 5.1 In-Domain Results · p. 7
Replacing question embeddings with an all-ones vector reduces AUC and ACC relative to full SQKT.
“Using question indicators (row 2) reduces AUC and ACC, highlighting the importance of the actual content of student questions and its effective utilization.”
Quote in the paperSubstantial claim - Findingown data§ 5.2 Cross-Domain Results · p. 8
In content structure generalization, the full SQKT model showed an absolute 45.3% improvement in AUC over the model without question data.
“Our full model (orange) showed an absolute 45.3% improvement in AUC over without using question data (blue).”
Quote in the paperSubstantial claim - Findingown data§ 5.2 Cross-Domain Results · p. 8
In data-scarce generalization, finetuning SQKT directly on the 'Algorithms' data shows an AUC score close to random.
“finetuning SQKT directly on the 'Algorithms' data (green) shows an AUC score close to random.”
Quote in the paperSubstantial claim - Findingown data§ 1 Introduction · p. 2
Integrating students' questions into KT enables more accurate predictions of students' success or failure on subsequent problems.
“To the best of our knowledge, this is the first work to integrate students' questions into KT, enabling more accurate predictions of students' success or failure on subsequent problems.”
Quote in the paperSubstantial claim- Rests on an assumption: No existing KT model incorporates students' questions.
- Findingown data§ 4.2 Experimental Setup · p. 6
The model struggled on the Algorithm course due to the small data size.
“This choice was motivated by our initial observation that the model struggled on this course due to the small data size.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.2 Cross-Domain Results · p. 8
Student questions convey generalizable insights into student performance across different courses.
“Both experiments conclude that student questions convey generalizable insights into student performance across different courses”
Quote in the paperSubstantial claim - Interpretationown data§ 5.1 In-Domain Results · p. 7
Skills and questions have a synergistic contribution to SQKT's performance.
“Relying solely on either skills (row 3) or questions (row 4) is suboptimal, demonstrating their synergistic contribution.”
Quote in the paperSubstantial claim - Premisecitation§ Abstract · p. 1
Students' questions often contain valuable signals about their understanding and misconceptions.
“Although students' questions often contain valuable signals about their understanding and misconceptions, traditional KT models often neglect to incorporate these questions as inputs to address these challenges.”
- Also stated in: § 1 Introduction · p. 1 · plain
“A valuable source of signals about students' understanding and misconceptions is the questions they ask (Sun et al., 2021).”
- Also stated in: § 2 Related Works · p. 2 · plain
“Questions arising from these interactions often reveal students' reasoning processes and areas of struggle in applying theoretical knowledge to coding (Sun et al., 2021).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Sun et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Also stated in: § 1 Introduction · p. 1 · plain
- Interpretationown data§ 1 Introduction · p. 2
The combination of student questions with dynamically extracted skill information enables more accurate and granular modeling of student knowledge states.
“The combination of student questions with dynamically extracted skill information enables more accurate and granular modeling of student knowledge states.”
Quote in the paperSubstantial claim - Interpretationnone given§ Abstract · p. 1
SQKT creates semantically rich embeddings that capture not only the surface-level content of questions but also the student's mastery level and conceptual understanding.
“Our method creates semantically rich embeddings that capture not only the surface-level content of the questions but also the student's mastery level and conceptual understanding.”
Quote in the paperNot a substantial claim - Premisenone given§ Abstract · p. 1
Traditional KT models often neglect to incorporate students' questions as inputs.
“Although students' questions often contain valuable signals about their understanding and misconceptions, traditional KT models often neglect to incorporate these questions as inputs to address these challenges.”
- Also stated in: § 1 Introduction · p. 1 · plain
“However, traditional KT models overlook this rich source of information.”
- Also stated in: § 2 Related Works · p. 2 · plain
“Yet, most existing models fail to capture the valuable insights embedded in students' questions.”
Quote in the paperSubstantial claim - Also stated in: § 1 Introduction · p. 1 · plain
- Interpretationreasoning§ 5.1 In-Domain Results · p. 7
Explicitly identifying skills from student questions further enhances the clarity of student performance.
“Further, the additional step of explicitly identifying skills from their questions appears to further enhance the clarity of student performance.”
Quote in the paperSubstantial claim - Interpretationown data§ 6 Conclusion · p. 8
SQKT integrates students' questions and auto-extracted skill information to provide a more comprehensive view of student knowledge than traditional KT models.
“By integrating students' questions and auto-extracted skill information, SQKT provides a more comprehensive view of student knowledge than traditional KT models.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5.1 In-Domain Results · p. 7
Question embeddings can be integrated with general KT models to enhance their predictive accuracy.
“It also suggests that our question embeddings can be integrated with general KT models to enhance their predictive accuracy.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.1 In-Domain Results · p. 7
SQKT is highly effective in predicting students' future performance.
“These results demonstrate that our SQKT model, which incorporates student questions, is highly effective in predicting students' future performance.”
Quote in the paperSubstantial claim - Interpretationown data§ 1 Introduction · p. 2
SQKT's strong performance in cross-domain settings highlights its generalizability across different course materials and environments.
“Our model's strong performance in cross-domain settings highlights its generalizability across different course materials and environments.”
Quote in the paperSubstantial claim - Study design and conductreasoning§ Limitations · p. 9
The skill extractor system employs a rule-based methodology rather than statistical machine learning techniques.
“Second, the skill extractor system employs a rulebased methodology rather than statistical machine learning techniques.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.1 In-Domain Results · p. 7
The superior performance of SQKT stems from unique insights provided by student questions, such as their understanding of theoretical concepts and specific struggles, which are not always apparent in code submissions alone.
“The results suggest that the superior performance of SQKT stems from the unique insights provided by student questions, such as their understanding of theoretical concepts and specific struggles, which are not always apparent in code submissions alone.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 3.1 Multi-feature Inputs · p. 4
Combining extracted skills with those required for the target problem allows the model to predict a student's performance more accurately.
“By combining these extracted skills with those required for the target problem, the model can predict a student's performance more accurately.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ 3.1 Multi-feature Inputs · p. 4
Extracting skills from student questions is more straightforward and accurate than from submitted codes, as these questions often directly address the concepts students find challenging.
“Extracting skills from student questions is more straightforward and accurate than from submitted codes, as these questions often directly address the concepts students find challenging.”
Quote in the paperNot a substantial claim - Interpretationown data§ 5.3 Error Analysis · p. 8
Complexity errors often occur because code snippets contain mixed language syntax, which challenges the model's parsing capabilities.
“often due to code snippets containing mixed language syntax, which challenges the model's parsing capabilities.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ 1 Introduction · p. 2
SQKT's performance boost stems from the questions and the automatically extracted skill information, which offer insights into conceptual understanding and reasoning processes that are difficult to capture from code submissions alone.
“Our analysis reveals that this performance boost stems from the questions and the automatically extracted skill information, which offer insights into conceptual understanding and reasoning processes that are difficult to capture from code submissions alone.”
Quote in the paperSubstantial claim - Premisecitation§ 2 Related Works · p. 2
Traditional KT models often use the Q-matrix method to manually tag problems with the required skills.
“Traditional KT models often use the Q-matrix method to manually tag problems with the required skills (Yu et al., 2022), but this process is labor-intensive and often fails to capture the full range of skills students use.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Yu et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisecitation§ 3.1 Multi-feature Inputs · p. 4
Incorporating error types as skills is motivated by the pedagogical principle that errors reveal students' understanding and misconceptions, which are correlated with learning gaps.
“Incorporating error types as skills was motivated by the pedagogical principle that errors reveal students' understanding and misconceptions, which are correlated with learning gaps (Altadmri and Brown, 2015; Becker et al., 2019; Hertz and Ford, 2013).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Altadmri and Brown, 2015
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Becker et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Hertz and Ford, 2013
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationown data§ 3.1 Multi-feature Inputs · p. 4
The skill extractor produces reliable outputs that closely match human judgments.
“These results indicate that the skill extractor produces reliable outputs that closely match human judgments.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 3.1 Multi-feature Inputs · p. 4
Fine-tuning CodeT5 with an auxiliary task of generating potential educator responses helps the question embedding capture the gist of a student's question that is predictive of the educator's response.
“This task helps the question embedding capture the gist of a student's question (mainly confusion and erroneous code) that is predictive of the educator's response.”
Quote in the paperNot a substantial claim - Interpretationown data§ Abstract · p. 1
SQKT leverages students' questions and automatically extracted skill information to enhance the accuracy of predicting students' performance on subsequent problems in programming education.
“This paper introduces SQKT (Students' Question-based Knowledge Tracing), a knowledge tracing model that leverages students' questions and automatically extracted skill information to enhance the accuracy of predicting students' performance on subsequent problems in programming education.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.1 In-Domain Results · p. 7
SQKT's architecture leverages student questions more effectively than the baselines.
“However, SQKT consistently outperformed these models, underscoring the efficacy of its architecture in leveraging student questions more effectively than the baselines.”
Quote in the paperSubstantial claim - Interpretationown data§ 5.1 In-Domain Results · p. 7
The question loss enriches the embedding space by capturing important information in student questions better.
“This loss seems to enrich the embedding space by capturing important information in student questions better.”
Quote in the paperNot a substantial claim - Premisereasoning§ 2 Related Works · p. 2
A significant gap remains in leveraging student-educator interactions.
“However, a significant gap remains in leveraging student-educator interactions.”
Quote in the paperNot a substantial claim - Interpretationown data§ 5.3 Error Analysis · p. 8
Confusion errors typically occur when the error in the code is unrelated to the student's question, making it difficult for the model to establish the correct correlation.
“typically occurring when the error in the code is unrelated to the student's question, making it difficult for the model to establish the correct correlation.”
Quote in the paperNot a substantial claim - Premisecitation§ 1 Introduction · p. 1
In programming education contexts, students' questions offer clearer insights into their understanding and confusion.
“In such contexts, students' questions offer clearer insights into their understanding and confusion (King, 1994).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites King, 1994
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning§ 5.1 In-Domain Results · p. 8
The triplet loss ensures effective integration of diverse data sources.
“The triplet loss ensures effective integration of diverse data sources.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 3.1 Multi-feature Inputs · p. 4
Identifying the skills students struggle with can improve model performance compared to relying solely on questions.
“Identifying the skills students struggle with can improve our model's performance compared to relying solely on questions.”
Quote in the paperNot a substantial claim - Interpretationown data§ 5.2 Cross-Domain Results · p. 8
Leveraging student questions greatly enhances the model's ability to adapt to new courses with varying difficulty levels and limited data.
“and that leveraging them greatly enhances the model's ability to adapt to new courses with varying difficulty levels and limited data.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 6 Conclusion · p. 8
SQKT captures valuable information about students' programming competencies through their questions.
“SQKT shows its ability to capture valuable information about students' programming competencies through their questions.”
Quote in the paperSubstantial claim
- Probably not a substantial claim. 19
- It may state two separate claims as one. 20
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Premisecitation§ 2 Related Works · p. 2Shi et al. (2022) introduced Code-DKT, which uses attention mechanisms to extract domain-specific code features.
“Shi et al. (2022) introduced Code-DKT, which uses attention mechanisms to extract domain-specific code features.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Shi et al. (2022) source
It may state two separate claims as one.
Findingown data§ 5.1 In-Domain Results · p. 7SQKT achieved an AUC of 87.1-93.4, representing an absolute improvement of 12.6-20.8 compared to the best-performing baseline (KTMFF+).
“SQKT achieved an AUC of 87.1-93.4, representing an absolute improvement of 12.6-20.8 compared to the best-performing baseline (KTMFF+).”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown data§ 5.1 In-Domain Results · p. 7In the ablation study reported in Table 3, SQKT achieved AUC 93.4%, ACC 89.2%, and F1 88.4%.
“Table 3: Ablation study on the 'Python Introduction' course.”
- Also stated in: § 5.1 In-Domain Results · p. 7 · plain
“| SQKT | 93.4 | 89.2 | 88 . 4 |”
Quote in the paperSubstantial claim- Also stated in: § 5.1 In-Domain Results · p. 7 · plain
It may state two separate claims as one.
Findingown data§ 6 Conclusion · p. 8SQKT consistently outperforms baseline models in both in-domain and cross-domain settings across various programming courses and difficulty levels.
“We demonstrate the effectiveness of SQKTacross various programming courses and difficulty levels, consistently outperforming baseline models in both in-domain and cross-domain settings.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingreasoning§ 3.1 Multi-feature Inputs · p. 4The auxiliary educator-response generation task enhances question embeddings and the model's overall prediction accuracy.
“This enhances question embeddings and the model's overall prediction accuracy.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Findingnone given§ 5.2 Cross-Domain Results · p. 8The 'Algorithm' course has small data.
“Since the 'Algorithm' course has small data”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown data§ 3.1 Multi-feature Inputs · p. 4The skill extractor achieved a precision of 0.85, a recall of 0.88, and an F1-score of 0.86 on a random sample of 100 student questions from the 'Python Basic' course.
“The skill extractor achieved a precision of 0.85, a recall of 0.88, and an F1-score of 0.86.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown data§ Abstract · p. 1SQKT exhibited robust generalization capabilities in cross-domain settings, effectively addressing data scarcity issues in advanced programming courses.
“The model also exhibited robust generalization capabilities in cross-domain settings, effectively addressing data scarcity issues in advanced programming courses.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
Findingown data§ 3.1 Multi-feature Inputs · p. 3Students' questions typically include two types of information: natural language questions that seek clarification on specific concepts or strategies, and code-based questions that address specific lines of code or errors.
“As illustrated by an example student-educator interaction in Figure 3, students' questions typically include two types of information: natural language questions that seek clarification on specific concepts or strategies (A), and code-based questions that address specific lines of code or errors (C).”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown data§ 5.1 In-Domain Results · p. 8The triplet loss has a stronger impact than the question loss, making a notable contribution especially for the 'First Python' course.
“The triplet loss, designed to unify the embedding space for heterogeneous input features, has stronger impact, making a notable contribution especially for the 'First Python' course.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown data§ 5.2 Cross-Domain Results · p. 8In data-scarce generalization, the full SQKT model showed an 11.4% improvement over the in-domain model and 16.7% over the cross-domain model incorporating no student questions.
“However, our full model (orange) showed a substantial improvement of 11.4% over the in-domain model (green) and 16.7% over the cross-domain model incorporating no student questions (blue).”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Premisecitation§ 2 Related Works · p. 2Manual tagging of skills is labor-intensive and often fails to capture the full range of skills students use.
“Traditional KT models often use the Q-matrix method to manually tag problems with the required skills (Yu et al., 2022), but this process is labor-intensive and often fails to capture the full range of skills students use.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Interpretationnone given§ 3.1 Multi-feature Inputs · p. 5The fusion layer enhances SQKT's ability to process heterogeneous yet semantically and contextually related signals more coherently.
“Consequently, the fusion layer enhances SQKT's ability to process heterogeneous yet semantically and contextually related signals more coherently.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
Interpretationreasoning§ 2 Related Works · p. 2The skill extraction method captures aspects of student knowledge that are not evident in code submissions alone, improving the prediction of student performance.
“This skill extraction method captures aspects of student knowledge that are not evident in code submissions alone, improving the prediction of student performance.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
Premisecitation§ 2 Related Works · p. 2Liu et al. (2022) developed an approach that considers the multi-skill nature of programming exercises by learning features from student code that reflect multiple skills.
“Liu et al. (2022) developed an approach that considers the multi-skill nature of programming exercises by learning features from student code that reflect multiple skills.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Liu et al. (2022)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim.
Study design and conductreasoning§ Limitations · p. 8The study did not apply any preprocessing to the input questions prior to analysis.
“First, we did not apply any preprocessing to the input questions prior to analysis.”
Quote in the paperSubstantial claimProbably not a substantial claim.
Premisereasoning§ Abstract · p. 1Knowledge tracing in programming education presents unique challenges due to the complexity of coding tasks and the diverse methods students use to solve problems.
“Knowledge tracing (KT) in programming education presents unique challenges due to the complexity of coding tasks and the diverse methods students use to solve problems.”
- Also stated in: § 2 Related Works · p. 2 · plain
“Programming education poses unique challenges for KT due to the complexity of coding tasks and multiple correct solutions that can be derived using various skills.”
Quote in the paperSubstantial claim- Also stated in: § 2 Related Works · p. 2 · plain
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
Premisenone given§ 3.1 Multi-feature Inputs · p. 4Traditional approaches rely on experts to manually tag skills for each problem, which is labor-intensive and lacks scalability.
“Traditional approaches rely on experts to manually tag skills for each problem, which is labor-intensive and lacks scalability.”
Quote in the paperNot a substantial claimProbably not a substantial claim.
Study design and conductreasoning§ Limitations · p. 8The skill extractor system was implemented to extract information from noisy inputs.
“To address this, we implemented a skill extractor system designed to effectively extract information from such noisy inputs.”
Quote in the paperSubstantial claim- Rests on an assumption: Student question inputs are often noisy.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
Premisecitation§ 2 Related Works · p. 2Recent research has advanced KT by incorporating behavioral data such as response times, scaffolding interactions, and attempt counts.
“Building upon the foundational approaches like Bayesian Knowledge Tracing (BKT) (Corbett and Anderson, 1994) and Deep Knowledge Tracing (DKT) (Piech et al., 2015), recent research has advanced KT by incorporating behavioral data, such as response times (Song et al., 2021), scaffolding interactions (Asselman et al., 2020), and attempt counts (Sun et al., 2022).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Song et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Asselman et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Sun et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
It may state two separate claims as one.
Premisereasoning§ 1 Introduction · p. 1The gap in using student questions is particularly significant in programming education, where KT is challenging because students' competencies need to be assessed from unstructured and noisy source code.
“This gap is particularly significant in programming education, where KT is challenging because students' competencies need be assessed from unstructured and noisy source code.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Interpretationreasoning§ 1 Introduction · p. 1Enriching embeddings with educator responses and skill information auto-extracted by GPT from students' questions creates a more comprehensive and detailed representation of the student's understanding, leading to improved prediction accuracy.
“Hence, our model enriches embeddings with two auxiliary signals: educator responses and skill information auto-extracted by GPT from students' questions. This approach creates a more comprehensive and detailed representation of the student's understanding, leading to improved prediction accuracy.”
Quote in the paperNot a substantial claimProbably not a substantial claim. It may state two separate claims as one.
Premisecitation§ 4.2 Experimental Setup · p. 6KTMFF and OKT are known for their strong performance in leveraging rich embeddings of code submissions and capturing the structural properties of code blocks.
“KTMFF and OKT are known for their strong performance in leveraging rich embeddings of code submissions and capturing the structural properties of code blocks.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Xiao et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Liu et al., 2022
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
Premisenone given§ 3.1 Multi-feature Inputs · p. 3Educator responses further clarify misconceptions, provide additional context, and highlight key concepts.
“Educator responses further clarify misconceptions, provide additional context, and highlight key concepts (B).”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
Interpretationnone given§ 2 Related Works · p. 2Using an automated skill-mapping system with GPT to extract knowledge components from student questions allows more flexible use of knowledge components, enabling the model to identify and leverage skills without extensive manual tagging.
“Our approach advances this by using an automated skill-mapping system using GPT to extract KCs from student questions. This method allows more flexible use of KCs, enabling the model to identify and leverage skills without extensive manual tagging.”
Quote in the paperNot a substantial claimProbably not a substantial claim. It may state two separate claims as one.
Study design and conductnone given§ 4.1 Dataset · p. 6The data for each course are split by students into training, validation, and test sets in an 8:1:1 ratio, with each student assigned to only one set to prevent the risk of information leakage.
“The data for each course are split by students into training, validation, and test sets in an 8:1:1 ratio, with each student assigned to only one set to prevent the risk of information leakage.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Study design and conductown data§ 4.3 Training Setup · p. 6There is no risk of a student's history being exposed to testing, as students do not overlap between the training and test sets.
“Note that there is no risk of a student's history being exposed to testing, as students do not overlap between the training and test sets (see Table 7).”
Quote in the paperSubstantial claimProbably not a substantial claim.
Interpretationreasoning§ Limitations · p. 9The rule-based choice aims to ensure interpretability, offering a clear and explainable mapping between questions and skills.
“This choice aims to ensure interpretability, offering a clear and explainable mapping between questions and skills.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
Premisereasoning§ 2 Related Works · p. 2The diversity in problem-solving approaches complicates tracing specific skills mastered by a student, making it challenging to predict future performance accurately.
“The diversity in problem-solving approaches complicates the tracing of specific skills mastered by a student, making it challenging to predict future performance accurately.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Premisenone given§ 4.2 Experimental Setup · p. 6The choice of baseline models is limited by the scarcity of prior KT models capable of processing code submissions without relying on predefined skill annotations.
“Our choice of baseline models is limited by the scarcity of prior KT models capable of processing code submissions without relying on predefined skill annotations.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Premisenone given§ 2 Related Works · p. 2These prior approaches still rely on manual tagging of knowledge components.
“However, these approaches still rely on manual tagging of KCs.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
No existing KT model incorporates students' questions.
“Moreover, to the best of our knowledge, no existing KT models incorporate student questions.”
“To the best of our knowledge, this is the first work to integrate students' questions into KT, enabling more accurate predictions of students' success or failure on subsequent problems.”
The paper relies on this without support.
The four Python courses in the dataset provide a diverse range of difficulty levels and topics.
“These data cover four distinct Python programming courses, providing a diverse range of difficulty levels and topics.”
The paper relies on this without support.
The skill extractor validated on the 'Python Basic' course generalizes to the other courses.
“To validate the skill extractor more systematically, we evaluated it on a random sample of 100 student questions from the 'Python Basic' course (§4.1).”
“We considered this level of accuracy acceptable, as extracted skills substantially improve SQKT's predictive performance (as discussed in the experiment section).”
The paper relies on this without support.
Student question inputs are often noisy.
“inputs are often noisy.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Table 3 is assigned to two different courses
MinorThe table caption says the ablation study is on Python Introduction, but the results discussion says Table 3 presents results on Python Basic.
“Table 3: Ablation study on the 'Python Introduction' course.”
“Table 3 presents the ablation results on the 'Python Basic' course (the same pattern is observed in other courses).”
Removing question loss improves First Python F1
MinorThe text says removing question loss causes slight performance drops across the three courses. In Table 4, First Python F1 instead rises from 84.9% to 86.9% when that loss is removed.
“The question loss, derived from the task of predicting educator responses to student questions, impacts performance across the three courses, with slight drops in performance when removed.”
“| SQKT | 92.3 86.3 87.3 93.0 87.1 84.9 93.3 88.4 89.8 | | - Question | 91.6 85.8 86.9 92.5 86.5 86.9 91.9 85.7 87.7 |”
Python Introduction SQKT AUC differs between ablation tables
MinorTable 3 reports 93.4% AUC for the full SQKT model on Python Introduction, whereas Table 4 reports 92.3% for SQKT on that course. No different condition is specified for the full-model rows.
“| SQKT | 93.4 | 89.2 | 88 . 4 |”
“| SQKT | 92.3 86.3 87.3 93.0 87.1 84.9 93.3 88.4 89.8 |”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Knowledge tracing: Modeling the acquisition of procedural knowledge
Corbett, A. T., & Anderson, J. R. (1995) · User Modelling and User-Adapted Interaction · doi:10.1007/bf01099821 · Cited by 2,324
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
King, A. (1994) · American Educational Research Journal · doi:10.3102/00028312031002338 · Cited by 406
Altadmri, A., & Brown, N. C. C. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677258 · Cited by 261
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243
Becker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199
Sun, D., Ouyang, F., Li, Y., & Zhu, C. (2021) · International Journal of STEM Education · doi:10.1186/s40594-021-00311-1 · Cited by 76
Dynamic Key-Value Memory Networks With Rich Features for Knowledge Tracing
Sun, X., Zhao, X., Li, B., Ma, Y., Sutcliffe, R. F. E., & Feng, J. H. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3051028 · Cited by 74
Open-ended Knowledge Tracing for Computer Science Education
Liu, N., Wang, Z., Baraniuk, R. G., & Lan, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.254 · Cited by 39
Liang, Y., Peng, T., Pu, Y., & Wu, W. (2022) · Scientific Reports · doi:10.1038/s41598-022-07956-0 · Cited by 33
Knowledge tracing based on multi-feature fusion
Xiao, Y., Xiao, R., Huang, N., Hu, Y., Li, H., & Sun, B. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07834-w · Cited by 18
Asselman, A., Khaldi, M., & Aammou, S. (2020) · Education and Information Technologies · doi:10.1007/s10639-019-10077-3 · Cited by 11
Lee, U., Bae, J., Jung, Y., Kang, M., Byun, G., Lee, Y. B., Kim, D., Lee, S., Park, J., Ahn, T., et al. (2024) · arXiv (Cornell University) · doi:10.13140/rg.2.2.25134.11847 · Cited by 6
Self-citationCode-DKT: A Code-based Knowledge Tracing Model for Programming Tasks
Shi, Y., Chi, M., Barnes, T., & Price, T. (2022) · arXiv · doi:10.48550/arxiv.2206.03545
Investigating factors of student learning in introductory courses
Hertz, M., & Ford, S. M. (2013) · Proceeding of the 44th ACM technical symposium on Computer science education · doi:10.1145/2445196.2445254
A Deep Knowledge Tracking Model Integrating Difficulty Factors
Song, Z., Huang, S., & Zhou, Y. (2021) · The 2nd International Conference on Computing and Data Science · doi:10.1145/3448734.3450792
Flagged first, then by citations
Research works printed but not linked: 3
- Allen B Downey and Chris Mayfield, editors. 2019. Think Java: How to think like a computer scientist . O'Reilly Media.
- Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge tracing. Advances in neural information processing systems , 28.
- Al Sweigart, editor. 2019. Automate the boring stuff with Python: practical programming for total beginners . No Starch Press.
Cited by
Sorted by citations
Code-driven programming prediction enhanced by LLM with a feature fusion approach
Liu, S., Li, J., Wan, Q., He, B., Huang, Z., & Li, Q. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104165 · Cited by 3
Vijayalakshmi, S., Prakash, M., & Subhasree, R. (2025) · International Conference on Image Information Processing (ICIIP) · doi:10.1109/iciip68302.2025.11346237
Ji, W., Wang, H., Wu, Q., & Zhou, G. (2026) · Journal Of Big Data · doi:10.1186/s40537-026-01385-w
GENERATING PEDAGOGICAL FEEDBACK FOR STUDENT CODE BASED ON LARGE LANGUAGE MODELS
Seliutin, D. (2026) · Collection of Scientific Publications NUS · doi:10.15589/znp2026.1(503).2.6
Wang, H., & Ji, W. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133050
Mahdaoui, M., Nouh, S., Alaoui, S. E. K., & Kandali, K. (2026) · Emerging Science Journal · doi:10.28991/esj-2026-010-05-032
Integration of Coding and Open-Ended Task Assessment for Ai-Based Evaluation of Student Competency
Yi, Z., & Chomsuwan, K. (2026) · International STEM Education Conference (iSTEM-Ed) · doi:10.1109/istem-ed69857.2026.11734512
Knowledge Graphs for Tracing Conceptual Understanding in Programming Education
Wang, T., & Zhang, D. (2026) · doi:10.1109/icaie70080.2026.11751382
Metadata
- Authors
- Kim, D., Kim, S., & Jo, Y.
- Affiliations
- Seoul National University (KR)
- ORCID
- Venue
- Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2025
- Pages
- 27703-27718
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W4412889816
- Cited by
- 9
- References
- 5
- Citations by year
- 2026: 8 · 2025: 1
- FWCI
- 3.31
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- National Research Foundation
- Korea Health Industry Development Institute
- National Research Foundation of Korea
- Ministry of Science and ICT, South Korea
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- knowledge tracing, programming education, student questions, skill extraction, Python programming, adaptive learning systems, domain generalization, data scarcity, misconceptions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
