CoChat
Sign Up
Cited by 7Intelligent Tutoring Systems and Adaptive LearningTeaching and Learning ProgrammingOnline Learning and Analytics

The DOI matches its DataCite record; 22 of the 22 references it cites were found in a registry or an index; 14 of the 30 sources its claims cite were checked for support.

Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks

Open access
Shi, Yang, Chi, Min, Barnes, Tiffany, Price, Thomas·2022·Zenodo
DOI 10.5281/zenodo.6853105
Open this paper in CoChat — free account

Abstract

Knowledge tracing (KT) models are a popular approach for predicting students' future performance at practice problems using their prior attempts. Though many innovations have been made in KT, most models including the state-of-the-art Deep KT (DKT) mainly leverage each student's response either as correct or incorrect, ignoring its content. Though many innovations have been made in KT, most models including the state-of-the-art Deep KT (DKT) mainly leverage each student's response either as correct or incorrect, ignoring its content. In this work, we propose Code-based Deep Knowledge Tracing (Code-DKT), a model that uses an attention mechanism to automatically extract and select domain-specific code features to extend DKT. We compared the effectiveness of Code-DKT against Bayesian and Deep Knowledge Tracing (BKT and DKT) on a dataset of 50 introductory programming problems, across 5 assignments. Our results show that Code-DKT consistently outperforms DKT by 3.07-4.00\% AUC across the 5 assignments, a comparable improvement to other state-of-the-art domain-general KT models over DKT. Finally, we analyze problem-specific performance and a set of case studies for one assignment to demonstrate when and how code features improve the Code-DKT's predictions.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

Code-DKT extends Deep Knowledge Tracing by using code2vec-style AST path embeddings and attention to incorporate students’ submitted code, and it consistently improves DKT on an introductory Java programming dataset. Across five assignments it raises AUC by 3.07–4.00 percentage points, though overall performance remains modest.

Key findings

  • On 410 students, 50 problems, and 5 assignments, Code-DKT outperformed DKT on every assignment: A1 74.31% vs 71.24%, A2 76.56% vs 73.09%, A3 80.40% vs 76.84%, A4 72.75% vs 69.16%, and A5 79.14% vs 75.14% AUC (3.07–4.00 point gains).
  • On assignment A1, Code-DKT (74.31% overall; 75.74% first attempts) beat BKT (63.78%; 50.22%), DKT-TFIDF (69.94%; 72.77%), and DKT-Expert (69.52%; 69.53%); the simple code-feature DKT baselines did not improve over DKT overall.
  • Problem-level A1 analysis showed Code-DKT outperformed DKT on 6 of 9 problems, ranging from +15.54% AUC (problem 13) to −4.43% (problem 236); gains appeared on problems sharing code structures/concepts, while losses involved unique concepts.
  • Ablation on A1: final Code-DKT 74.31% AUC; correctness only in attention 73.81%; correctness only in trace 73.84%; RNN 73.63%; static pretrained embedding 68.74%. In one case study, Code-DKT made 11/16 correct predictions vs DKT’s 8/16.

Methods

  • Dataset: introductory Java programming class at a large US university, Spring 2019, stored in ProgSnap2; 410 students, 50 problems across 5 assignments; 23.68% of attempts were correct; submissions were auto-graded with test cases, with correctness only when all tests passed.
  • Model: Code-DKT combines DKT (RNN/LSTM) with code2vec; code submissions are represented as AST leaf-to-leaf paths, embedded and attention-weighted using current/past correctness scores, then concatenated with DKT problem-correctness vectors for final LSTM prediction.
  • Evaluation: per assignment, 4:1 train/test split, 10 repeated runs; hyperparameters tuned on validation (embedding size 300, learning rate 0.0005, 40 epochs, max trace length 50); AUC was the primary metric; baselines were BKT, DKT, DKT-TFIDF, and DKT-Expert.

Limitations

  • Stated by authors: all models including Code-DKT had relatively low performance, partly due to problem difficulty and small data (410 students); the dataset came from a single semester/course, so generalization to other semesters, classes, or programming languages is unclear; only DKT was used as the baseline to extend and compare, so code features may affect other models differently.
  • Evident in the text: the evaluation is limited to one small, single-course Java dataset with class imbalance (23.68% correct) and many attempts per problem (6.1 average); it tests one code representation (code2vec), and the case-study mechanisms for why code features help are presented as hypotheses rather than verified explanations.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.5281/zenodo.6853105

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

14 of 30 cited sources checked · 6 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 6. DISCUSSION · p. 9

    DKT has historically been effective on other datasets, which are both larger and in other domains, such as EdNet, Assistments, and KhanAcademy.

    “By contrast, DKT has historically been effective on other datasets, which are both larger and in other domains, such as EdNet [13], Assistments [41] and KhanAcademy [34].”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [13] [13] Y. Choi, Y. Lee, D. Shin, J. Cho, S. Park, S. Lee, J. Baek, C. Bae, B. Kim, and J. Heo. Ednet: A large-scale hierarchical dataset in education. In International Conference on Artificial Intelligence in Education , pages 69-73. Springer, 2020. - source

      The Introduction identifies EdNet as a large dataset of TOEIC-preparation interactions, supporting its use as the example attached to this citation. Our copy does not report that DKT was effective on EdNet; a search for DKT found no performance result. The separately cited Assistments and KhanAcademy examples are outside this verdict. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [41] [41] D. Selent, T. Patikorn, and N. Heffernan. Assistments dataset from multiple randomized controlled experiments. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale , pages 181-184, 2016. - source

      The Contributions section identifies the paper as presenting an ASSISTments dataset, and the ASSISTments and Data Description sections describe its mathematics-learning domain and size (8,297 students and 102,252 attempted problems); page numbers are unavailable in our copy. The paper does not report that DKT was effective on the dataset or establish that it was larger than the citing paper’s dataset. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [34] [34] C. Piech, J. Bassen, J. Huang, S. Ganguli, M. Sahami, L. J. Guibas, and J. Sohl-Dickstein. Deep knowledge tracing. Advances in Neural Information Processing Systems , 28, 2015. -
  • Interpretationcitation
    § 2.2 Student Modeling in CS Education · p. 3

    Deep learning methods for code analysis have shown feasibility for similar usage on KT tasks.

    “All these methods reported better performance than traditional data-driven models on their tasks, showing the feasibility of similar usage on KT tasks.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [19] [19] R. Gupta, A. Kanade, and S. Shevade. Neural attribution for semantic bug-localization in student programs. Advances in Neural Information Processing Systems , 32, 2019. -
    • cites [42] [42] Y. Shi, Y. Mao, T. Barnes, M. Chi, and T. W. Price. More with less: Exploring how to use deep learning effectively through semi-supervised learning for automatic bug detection in student code. In In Proceedings of the 14th International Conference on Educational Data Mining (EDM) 2021 , 2021. -
    • cites [43] [43] Y. Shi, K. Shah, W. Wang, S. Marwan, P. Penmetsa, and T. Price. Toward semi-automatic misconception discovery using code embeddings. In LAK21: 11th International Learning Analytics and Knowledge Conference , pages 606-612, 2021. - source

      Section 5.1 (page not provided) reports that the code2vec model outperformed all baselines when predicting students’ performance on a programming exercise. The paper does not test knowledge-tracing (KT) tasks, so feasibility for similar use on KT tasks remains an extrapolation rather than a finding of this source. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Interpretationcitation
    § Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks · p. 1

    The improvement of Code-DKT over DKT is comparable to that of other modern KT models over DKT (2-4% AUC).

    “This improvement is comparable to that of other modern KT models over DKT (2-4%) [31, 44]”
    • Also stated in: § 5.1.1 Code-DKT vs DKT · p. 6 · plain
      “this improvement is comparable to SAINT+'s improvement over DKT on EdNet (+2 . 76%) [13], or SAKT's improvement on various datasets (+3.8%)”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites [31] [31] S. Pandey and G. Karypis. A self-attentive model for knowledge tracing. In In Proceedings of the 12th International Conference on Educational Data Mining (EDM) 2019 , 2019. - source
    • cites [44] [44] D. Shin, Y. Shim, H. Yu, S. Lee, B. Kim, and Y. Choi. Saint+: Integrating temporal features for ednet correctness prediction. In LAK21: 11th International Learning Analytics and Knowledge Conference , pages 490-496, 2021. - source
    • cites [13] [13] Y. Choi, Y. Lee, D. Shin, J. Cho, S. Park, S. Lee, J. Baek, C. Bae, B. Kim, and J. Heo. Ednet: A large-scale hierarchical dataset in education. In International Conference on Artificial Intelligence in Education , pages 69-73. Springer, 2020. - source

      The Introduction describes EdNet as a dataset for tasks including knowledge tracing, and its Table 1 compares dataset properties, not model performance. Searches for DKT comparisons and AUC found no passage reporting that modern KT models improve on DKT by 2–4%. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Interpretationreasoning
    § Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks · p. 1

    Domain-specific features may be just as important as model structure for improving KT performance.

    “suggesting that domain-specific features may be just as important as model structure.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [31] [31] S. Pandey and G. Karypis. A self-attentive model for knowledge tracing. In In Proceedings of the 12th International Conference on Educational Data Mining (EDM) 2019 , 2019. - source

      The Introduction says SAKT uses relevant knowledge concepts from past interactions and reports improved KT performance. But the paper does not compare the importance of domain-specific features with model structure; its ablation study varies architectural components, not that trade-off. No page numbers are available in our copy. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [44] [44] D. Shin, Y. Shim, H. Yu, S. Lee, B. Kim, and Y. Choi. Saint+: Integrating temporal features for ednet correctness prediction. In LAK21: 11th International Learning Analytics and Knowledge Conference , pages 490-496, 2021. - source

      Section 5.4 reports that adding temporal features to SAINT improved knowledge-tracing accuracy and AUC, supporting the importance of these features for performance. The paper does not establish that features are as important as model structure; its comparison with SAINT does not measure their relative importance. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • Findingown data
    § 5.1.2 Code-DKT vs Naive Code Features and BKT · p. 6

    On assignment A1, simple features derived from student code negatively affect the overall performance of DKT.

    “These simple features derived from student code instead negatively affect overall performance.”
    Quote in the paperSubstantial claim
  • It may state two separate claims as one. 38
  • Probably not a substantial claim. 32

  • It may state two separate claims as one.

    Premisecitation
    § 6. DISCUSSION · p. 10

    Code-DKT's improvement is comparable to SAINT over DKT on EdNet (+2.76% AUC) and SAKT over DKT on various datasets (+3.8%).

    “This is comparable to the improvement of SAINT over DKT on the EdNet dataset (they achieved +2.76% in AUC), or SAKT over DKT on various datasets (+3.8%) [31].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [31] [31] S. Pandey and G. Karypis. A self-attentive model for knowledge tracing. In In Proceedings of the 12th International Conference on Educational Data Mining (EDM) 2019 , 2019. - source

      Table 3 reports average AUCs of 0.824 for SAKT and 0.786 for DKT across five datasets, a difference of 0.038 (3.8 percentage points). Our copy does not report a SAINT comparison on EdNet or the claimed +2.76% AUC improvement. (LLM, conf=0.90)

      Experimental — accuracy not yet measured

  • It may state two separate claims as one.

    Premisecitation
    § 3.1 Deep Knowledge Tracing · p. 4

    Recent models have outperformed DKT by only about 2-4% AUC, suggesting DKT is still representative of modern deep KT models.

    “Some recent models have outperformed DKT, but only by about 2-4% AUC [31, 44], suggesting that DKT is still representative of modern deep KT models.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [31] [31] S. Pandey and G. Karypis. A self-attentive model for knowledge tracing. In In Proceedings of the 12th International Conference on Educational Data Mining (EDM) 2019 , 2019. - source

      Table 3 (listed under Conclusion and Future Work; no page given) shows SAKT exceeding DKT by about 2–4 AUC percentage points on ASSIST2009 and STATICS. But the gain is much larger on ASSIST2015 (0.854 versus 0.736), so the source does not support treating 2–4% as a general upper bound. The statement that DKT remains representative is the citing authors’ inference. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
    • cites [44] [44] D. Shin, Y. Shim, H. Yu, S. Lee, B. Kim, and Y. Choi. Saint+: Integrating temporal features for ednet correctness prediction. In LAK21: 11th International Learning Analytics and Knowledge Conference , pages 490-496, 2021. - source

  • It may state two separate claims as one.

    Study design and conductown data
    § 4.3 Baselines · p. 6

    AUC is a standard evaluation metric for KT models and is more appropriate than accuracy for imbalanced datasets like ours.

    “Our primary performance metric is AUC, a standard evaluation metric for KT models [34, 44, 31], as it uses the predicted probability of success, rather than a binary correctness prediction, and is more appropriate than accuracy for imbalanced datasets like ours (23% positive).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [34] [34] C. Piech, J. Bassen, J. Huang, S. Ganguli, M. Sahami, L. J. Guibas, and J. Sohl-Dickstein. Deep knowledge tracing. Advances in Neural Information Processing Systems , 28, 2015. -
    • cites [44] [44] D. Shin, Y. Shim, H. Yu, S. Lee, B. Kim, and Y. Choi. Saint+: Integrating temporal features for ednet correctness prediction. In LAK21: 11th International Learning Analytics and Knowledge Conference , pages 490-496, 2021. - source
    • cites [31] [31] S. Pandey and G. Karypis. A self-attentive model for knowledge tracing. In In Proceedings of the 12th International Conference on Educational Data Mining (EDM) 2019 , 2019. - source

  • It may state two separate claims as one.

    Interpretationcitation
    § 3.2 Deep Code Knowledge Tracing · p. 4

    Code-DKT extends code2vec for code classification, encoding an AST using a set of leaf-to-leaf paths.

    “Code-DKT extends the code2vec model for code classification, which encodes an AST using a set of leaf-to-leaf paths throughout the AST.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [5] [5] U. Alon, M. Zilberstein, O. Levy, and E. Yahav. code2vec: Learning distributed representations of code. Proceedings of the ACM on Programming Languages , 3(POPL):1-29, 2019. - source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § 5.2 Case Studies · p. 9

    The same direct usage of Boolean variables is seen in the if condition and return statement of Problem 5.

    “This same direct usage of Boolean variables is seen in the if condition and return statement of Problem 5.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § 4.1 Dataset & Experiments Setup · p. 5

    Typical solutions for the problems ranged from 10 to 20 lines of code.

    “typical solutions ranged 10 to 20 lines of code.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 5.2 Case Studies · p. 8

    Sometimes Code-DKT is overconfident in these situations, and incorrectly predicts success, as in Case B.

    “Sometimes Code-DKT is overconfident in these situations, and incorrectly predicts success, as in Case B.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 5.1.2 Code-DKT vs Naive Code Features and BKT · p. 6

    On assignment A1, neither simple expert-authored code features nor TFIDF data-driven features improve the overall performance of DKT.

    “The results show that neither the simple expert features nor the TFIDF datadriven features improve the overall performance of DKT.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 6. DISCUSSION · p. 9

    On assignment A1, DKT fails to identify two thirds of unsuccessful attempts, and over half of the time when it predicts a failed attempt, the student actually succeeded.

    “so the model fails to identify two thirds of unsuccessful attempts, and over half of the time when the model predicts a failed attempt, the student actually succeeded.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 6. DISCUSSION · p. 9

    Code-DKT, which had far more parameters, performed better than DKT.

    “and our Code-DKT model, which had far more parameters, performed better.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § 5.1.3 When is Code-DKT Effective? · p. 7

    Problem 233 requires the Math.abs function.

    “and 233 requires the Math.abs function (which many students failed to use correctly).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingnone given
    § 4.1 Dataset & Experiments Setup · p. 5

    Students in the dataset tended to make multiple submissions before succeeding finally.

    “Students tended to make multiple submissions before succeeding finally”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 5.2 Case Studies · p. 9

    DKT's confidence on problem 5 only modestly increased from 41.8% to 47.3% to 47.0%, leading it to incorrectly predict failure.

    “However, DKT's confidence only modestly increased from 41.8% to 47.3% to 47.0%, leading it to incorrectly predict failure.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 5.2 Case Studies · p. 8

    Code-DKT makes 11 out of 16 successful predictions on the student's traces, while DKT makes 8 correct.

    “The heatmaps for the student (Figures 4 and 5) show that Code-DKT is able to make better predictions on the traces than DKT, making 11 out of 16 successful predictions, while DKT is able to make 8 of them correct.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 6. DISCUSSION · p. 9

    For assignment A1, DKT's recall was 31.4% and precision was 46.5%.

    “For example, for assignment A1, the recall of DKT was 31.4% and the precision was 46.5%”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § 4.1 Dataset & Experiments Setup · p. 5

    23.68% of the attempts in the dataset were correct.

    “23 . 68% of the attempts were correct.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 5.2 Case Studies · p. 9

    After seeing the almost-correct code at the student's 4th attempt on Problem 13, Code-DKT predicted that the student would succeed on the next attempt, but it took one extra attempt to get it right.

    “After seeing the almost-correct code at their 4th attempt, CodeDKT predicted that the student would succeed on the next attempt since the modifications they needed were minimal (just write ' return value; '), but it took one extra attempt to get it right.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 5.2 Case Studies · p. 9

    When the student successfully completes problems 232 and 233 in a single attempt, Code-DKT's prediction of the student's success on problem 5 increases from 42.0% to 50.0% to 72.5%, leading it to successfully predict success on the student's first attempt at problem 5.

    “when the student successfully completes problems 232 and 233 in a single attempt, Code-DKT's prediction of the student's success on problem 5 increases from 42.0% to 50.0% to 72.5% respectively, leading it successfully predict success on the student's first attempt at problem 5.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data
    § 4.3 Baselines · p. 6

    Using skill vectors instead of one-hot encoded problem IDs slightly improved performance on the validation dataset.

    “Testing on the validation dataset showed slightly improved performance using these skill vectors.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 6. DISCUSSION · p. 10

    Code-DKT's improvement is robust, with a +3% to +4% improvement overall on all five assignments.

    “Code-DKT's improvement is also robust. It has a +3% to +4% improvement overall on all five assignments.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 6. DISCUSSION · p. 10

    Most problems in this dataset did share primary learning concepts (e.g. loops, conditionals) and benefit from code features.

    “However, most problems in our dataset did share primary learning concepts (e.g. loops, conditionals) and benefit from code features”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 5.2 Case Studies · p. 8

    After the student's second incorrect attempt on Problem 235, Code-DKT's prediction of success probability increased from 44.2% to 49.1%, while DKT's prediction decreased from 47.7% to 46.7%.

    “its prediction of success probability increased from 44.2% to 49.1% after the student's second (incorrect) attempt, while DKT's prediction decreased from 47.7% to 46.7%.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingnone given
    § 6. DISCUSSION · p. 9

    In this dataset, problem descriptions were complex, solutions involved complex conditional logic, and students had to write perfect Java syntax for the program to compile.

    “In our dataset, the problem descriptions were complex, and their solutions involved complex conditional logic, and students had to write perfect Java syntax for the program to compile.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingreasoning
    § 6. DISCUSSION · p. 10

    All models evaluated, including Code-DKT, have relatively low performance, partially due to the difficulty of the problem and low data size (410 students).

    “All models evaluated, including Code-DKT, have a relatively low performance, partially due to the difficulty of the problem and low data size (410 students), as discussed in Section 6.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data
    § 7. CONCLUSION · p. 10

    The best baseline model performance was about 73%, and Code-DKT was 74.3%, demonstrating considerable room for improvement on modeling for knowledge tracing in programming.

    “However, the best baseline model performance was about 73%, and Code-DKT was 74.3%, demonstrating considerable room for improvement on modeling for knowledge tracing in programming.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data
    § 6. DISCUSSION · p. 10

    Code features are useful for predicting both first attempts and subsequent attempts.

    “We also found that code features are useful for predicting both first attempts and subsequent attempts.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationown data
    § 5.2 Case Studies · p. 9

    Both models know that problems 232, 233, and 5 are related and share some learning concepts, but Code-DKT's analysis of the student code allowed it to infer more about the knowledge demonstrated in past problems.

    “Both models know that these problems are related, and share some learning concepts (based on how other students' successes on the problems are related), but Code-DKT's analysis of the student code allowed it to infer more about the knowledge that was demonstrated in past problems.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Interpretationreasoning
    § 2.1 Knowledge Tracing · p. 2

    Code features can be used to make better knowledge tracing models.

    “We use these code features to make better knowledge tracing models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Interpretationreasoning
    § 2.2 Student Modeling in CS Education · p. 3

    Newer ITSs that support written code make knowledge tracing in computer science more challenging.

    “This provides better practice for students, but also makes knowledge tracing in computer science a more challenging task.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 2.2 Student Modeling in CS Education · p. 2

    Incorporating structural code features can improve DKT for a single problem from a large 'hour of code' dataset.

    “Like our current work, Wang et al. showed that incorporating structural code features can improve DKT for a single problem from a large 'hour of code' (HoC) dataset [52].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [52] [52] L. Wang, A. Sy, L. Liu, and C. Piech. Learning to represent student knowledge on programming exercises using deep learning. In In: Proceedings of the 10th International Conference on Educational Data Mining (EDM 2017) , 2017. -

  • Probably not a substantial claim. The authors may not put it forward as true. It may state two separate claims as one.

    Interpretationown data
    § 5.1.3 When is Code-DKT Effective? · p. 7

    Code-DKT's code features are most useful at predicting problems that share code structures with other problems, and less useful at predicting problems that emphasize novel code structures.

    “Together, these results suggest a hypothesis that Code-DKT's code features are most useful at predicting problems that share code structures with other problems, and less useful at predicting problems that emphasize novel code structures.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Premisenone given
    § 2.3 Deep Code Learning · p. 3

    Simple structural approaches are often outperformed by deep learning models, especially on big datasets.

    “However, these simple structural approaches are often outperformed by newly developed deep learning models, especially when applied to big datasets.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationcitation
    § 2.3 Deep Code Learning · p. 3

    Code-DKT's code extraction component is based on code2vec but adds score to the attention mechanism to assign weights to code paths for predictions.

    “Code-DKT's code extraction component is based on the code2vec model, but adds score to the attention mechanism to assign weights to code paths [4] for predictions.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [4] [4] U. Alon, M. Zilberstein, O. Levy, and E. Yahav. A general path-based representation for predicting program properties. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation , pages 404-419, 2018. - source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given
    § 4.3 Baselines · p. 6

    Code features such as else if statements and && operations represent students' usage of concepts like writing alternative conditions or using 'and' logic.

    “These statements and operations represent students' usage of certain concepts such as writing alternative conditions, or using 'and' logic to solve a problem.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationown data
    § 5.1.4 Ablation Study · p. 7

    Correctness information helps both in attending to relevant code paths and in final predictions.

    “suggesting that correctness information helps both in attending to relevant code paths, and final predictions, but this information is somewhat redundant.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationown data
    § 5.1.3 When is Code-DKT Effective? · p. 7

    Code features can reduce model performance, but the potential for Code-DKT's improvement seems greater than the potential for harm.

    “It also shows that code features can reduce model performance, but the potential for Code-DKT's improvement seems to be greater than the potential for harm.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 3.2 Deep Code Knowledge Tracing · p. 4

    Code-DKT's model structure diverges from code2vec to account for the specific needs of the KT problem.

    “The Code-DKT model structure then diverges somewhat from code2vec, to account for the specific needs of the KT problem.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisecitation
    § 2.3 Deep Code Learning · p. 3

    code2vec outperformed prior models on code classification.

    “Another recent model, code2vec, outperformed these models.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] [5] U. Alon, M. Zilberstein, O. Levy, and E. Yahav. code2vec: Learning distributed representations of code. Proceedings of the ACM on Programming Languages , 3(POPL):1-29, 2019. - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim.

    Study design and conductreasoning
    § 5.1.1 Code-DKT vs DKT · p. 6

    For each assignment, a new model was trained and tested separately without using data from prior assignments.

    “Note that for each assignment, a new model is trained and tested separately, without using data from prior assignments.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Interpretationnone given
    § 2.1 Knowledge Tracing · p. 2

    In programming education, students' code submissions contain rich information on the state of their current knowledge.

    “In the domain of programming education, for example, students' code submissions contain rich information on the state of their current knowledge.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Interpretationnone given
    § 2.3 Deep Code Learning · p. 3

    The attention mechanism learns weights for different features, allowing the model to directly use score information to select the most predictive code paths.

    “The attention mechanism learns weights for different features, allowing the model to directly use score information to select the most predictive code paths.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given
    § 5.2 Case Studies · p. 8

    Even with code features, there is still a great deal of uncertainty; no matter how close a student is to a correct answer, there is no guarantee they will achieve it on their next attempt.

    “Even with code features, there is still a great deal of uncertainty. No matter how close a student is to a correct answer, there is no guarantee they will achieve it on their next attempt.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given
    § 3.1 Deep Knowledge Tracing · p. 4

    DKT's LSTM structure makes it straightforward to extend with code features and to directly evaluate those features' contributions.

    “Further, its LSTM structure makes it straightforward to extend with code features and to directly evaluate those features' contributions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given
    § 3.2 Deep Code Knowledge Tracing · p. 5

    Not all parts of a student's code are relevant, so not all code paths are important for predicting future success.

    “However, not all parts of a student's code are relevant, and thus not all code paths e r are important for predicting a student's future success.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Interpretationown data
    § 6. DISCUSSION · p. 9

    To the extent that this dataset is representative of the domain, the results suggest the need for improved KT models for programming.

    “Thus, to the extent that our datasets is representative of the domain, our results suggest the need for improved KT models for programming.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationcitation
    § Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks · p. 1

    Code-DKT uses the code2vec model to learn a representation of student code and combines it with DKT to track student progress.

    “we introduce a novel code-based deep knowledge tracing (Code-DKT) model, which uses the code2vec model [4] to learn a meaningful representation of student code, and combines this with Deep Knowledge Tracing (DKT) [34] to track student progress.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [4] [4] U. Alon, M. Zilberstein, O. Levy, and E. Yahav. A general path-based representation for predicting program properties. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation , pages 404-419, 2018. - source
    • cites [34] [34] C. Piech, J. Bassen, J. Huang, S. Ganguli, M. Sahami, L. J. Guibas, and J. Sohl-Dickstein. Deep knowledge tracing. Advances in Neural Information Processing Systems , 28, 2015. -

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Premisenone given
    § 3. METHOD · p. 3

    Historically, KT algorithms have only utilized problem ID and correctness, and this work extends the input sequence to include submitted code.

    “Historically, KT algorithms have only utilized q t and a t , and in this work we extend the input sequence to include c t .”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Premisenone given
    § Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks · p. 1

    Little work has investigated whether domain-general KT models can predict student success in programming, or how domain-specific features might improve performance.

    “Little work has investigated whether domain-general KT models can predict student success in programming, or how domainspecific features might improve performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Interpretationown data
    § 5.2 Case Studies · p. 9

    The code submission on Problem 233 suggests the student is able to combine logical operators with Boolean variables to return a Boolean expression, which occurs again in the return statement of the student's attempt at Problem 5.

    “The code submission on Problem 233 further suggests the student is able to combine logical operators with Boolean variables to return a Boolean expression. This occurs again in the return statement of the students' attempt at Problem 5”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationreasoning
    § Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks · p. 1

    In programming, incorrect attempts can vary from minor syntax errors to clear misconceptions, and correct answers can reveal different levels of concept mastery depending on conciseness and concepts used.

    “In programming, for example, one incorrect attempt may have a minor syntax error while another includes a clear misconception. Similarly, two different correct answers could reveal dramatically different levels of concept mastery depending on their conciseness and the concepts used.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.

    Premisecitation
    § 3.1 Deep Knowledge Tracing · p. 4

    LSTM is widely perceived as an advancement over RNNs.

    “which is widely perceived as an advancement over RNNs.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] [21] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation , 9(8):1735-1780, 1997. - source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Premisenone given
    § 2.3 Deep Code Learning · p. 3

    Deep neural networks have achieved better performance than traditional data-driven methods in software engineering.

    “Deep neural networks have been applied in the software engineering domain, and achieved better performance than traditional data-driven methods.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationown data
    § 5.1.3 When is Code-DKT Effective? · p. 7

    The content of a student's code is helpful for predicting not only how quickly they will solve the current problem, but also future problems.

    “This suggests that the content of a student's code is helpful for not only predicting how quickly they will solve the current problem, but also future problems.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationnone given
    § 5.2 Case Studies · p. 9

    In Problem 232, the student directly uses a Boolean variable in the if-condition rather than a superfluous comparison, demonstrating a higher level of understanding.

    “For example, in Problem 232, the student directly uses Boolean variable in the if-condition ( if (vacation) ) rather than a superfluous comparison ( if (vacation == true) ) that many students use, demonstrating a higher level of understanding.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Interpretationnone given
    § 2.1 Knowledge Tracing · p. 2

    More structural information can be extracted from student code submissions to infer students' learning status of certain concepts.

    “As proposed in this paper, more structural information could be extracted from student code submission to infer students' learning status of certain concepts.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationown data
    § 6. DISCUSSION · p. 10

    Code-DKT's overall performance is still relatively poor, suggesting the need for more work on leveraging domain-specific features for improved KT.

    “Importantly, however this is still a relatively poor performance overall, suggesting the need for more work on leveraging domain-specific features for improved KT.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Interpretationreasoning
    § 6. DISCUSSION · p. 10

    For first attempts, the model seems to identify code structures in prior attempts that indicate knowledge or competence with certain programming concepts, which it uses to make predictions on new problems.

    “For first attempts, the model seems to identify code structures in prior attempts that indicate knowledge or competence with certain programming concepts, which it uses to make predictions on new problems.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationreasoning
    § 6. DISCUSSION · p. 10

    The models perform considerably better than chance, and such models could still be useful, e.g. in prioritizing help to struggling students.

    “Still, they perform considerably better than chance, and such models could still be useful, e.g. in prioritizing help to struggling students.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 6. DISCUSSION · p. 10

    For repeated attempts, the model seems to use the relative correctness of a student's code to determine how close they are to a solution and therefore how likely they are to get it right on the next attempt.

    “For repeated attempts, the model seems to use the relative correctness of a student's code to determine how close they are to a solution and therefore how likely they are to get it right on the next attempt.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may state two separate claims as one.

    Interpretationown data
    § 7. CONCLUSION · p. 10

    The case studies illustrate specific situations where knowledge tracing can be particularly difficult in programming, and where there is potential for improving code KT, e.g. when common code structures are used across problems.

    “The case studies in this paper illustrate specific situations where knowledge tracing can be particularly difficult in programming, and where there is potential for improving code KT, e.g. when common code structures are used across problems.”
    Quote in the paperSubstantial claim
60 of 60

  • The authors state this as an assumption.

    The latest submissions are more useful than earlier submissions.

    “assuming the latest submissions were more useful.”

  • The paper relies on this without support.

    First attempts are important in a KT task because they represent points at which an ITS might make key interventions.

    “First attempts are important in a KT task because they represent points at which an ITS might make key interventions (e.g. offering a worked example if a student might fail at problem solving).”

  • The paper relies on this without support.

    Relatively simple code features (e.g. the presence of a return statement) should be at least somewhat related to how close a student is to a correct answer.

    “This result is somewhat surprising, given that relatively simple features (e.g. the presence of a return statement) should be at least somewhat related to how close a student is to a correct answer.”

  • The paper relies on this without support.

    Many tutoring systems only have hundreds of students.

    “Regardless, many tutoring systems only have hundreds of students, and effective KT models must still be able to perform well on these small datasets.”

  • The paper relies on this without support.

    Repeated practice is a common feature of many CS1 courses.

    “and this repeated practice is a common feature of many CS1 courses.”

  • The paper relies on this without support.

    The path-based AST representation used by code2vec easily captures code structures.

    “These code structures are easily captured by the path-based AST representation used by code2vec.”

  • The paper relies on this without support.

    code2vec has the ability to recognize repeated patterns in ASTs.

    “given code2vec's ability to recognize repeated patterns in ASTs.”
7 of 7

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Code-path count conflicts with its indexing

    Minor

    Indexing paths from p 0 through p R denotes R+1 paths, while the same sentence says there are R paths.

  • Reported DKT AUC range excludes assignment A3

    Minor

    The stated upper bound of 75% excludes the 76.84% AUC reported for DKT on A3 in Table 1.

  • Case study gives two different counts of problem 13 attempts

    Minor

    The heatmap description says the student failed problem 13 four times before succeeding, making five attempts. Case B describes the sixth attempt as the final successful one.

  • Abstract reports 50 students, while the dataset description reports 410

    Minor

    The abstract describes a class of 50 students, but the introduction and dataset section both state that the dataset contains 410 students.

4 of 4

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 32

  1. [2][2] M. Allamanis, H. Peng, and C. Sutton. A convolutional attention network for extreme summarization of source code. In International Conference on Machine Learning , pages 2091-2100. PMLR, 2016. -
  2. [6][6] J. R. Anderson and B. J. Reiser. The lisp tutor. Byte , 10(4):159-175, 1985. -
  3. [7][7] A. Badrinath, F. Wang, and Z. Pardos. pybkt: An accessible python library of bayesian knowledge tracing models. In In: Proceedings of the 14th International Conference on Educational Data Mining (EDM 2021) . ERIC, 2021. -
  4. [9][9] T. Barnes. The q-matrix method: Mining student response data for knowledge. In American Association for Artificial Intelligence 2005 Educational Data Mining Workshop , pages 1-8. AAAI Press, Pittsburgh, PA, USA, 2005. -
  5. [17][17] T. Gervet, K. Koedinger, J. Schneider, T. Mitchell, et al. When is deep learning the best approach to knowledge tracing? Journal of Educational Data Mining , 12(3):31-54, 2020. -
  6. [19][19] R. Gupta, A. Kanade, and S. Shevade. Neural attribution for semantic bug-localization in student programs. Advances in Neural Information Processing Systems , 32, 2019. -
  7. [22][22] B. P. Iii, A. Hicks, and T. Barnes. Generating hints for programming problems using intermediate output. In In: Proceedings of the 7th International Conference on Educational Data Mining (EDM 2014) , 2014. -
  8. [23][23] W. Jin, L. Lehmann, M. Johnson, M. Eagle, B. Mostafavi, T. Barnes, and J. Stamper. Towards automatic hint generation for a data-driven novice programming tutor. In Workshop on Knowledge Discovery in Educational Data, 17th ACM Conference on Knowledge Discovery and Data Mining , 2011. -
  9. [24][24] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015 , 2015. -
  10. [26][26] Q. Liu, Z. Huang, Y. Yin, E. Chen, H. Xiong, Y. Su, and G. Hu. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering , 33(1):100-115, 2019. -
  11. [27][27] M. Maniktala, C. Cody, A. Isvik, N. Lytle, M. Chi, and T. Barnes. Extending the hint factory for the assistance dilemma: A novel, data-driven helpneed predictor for proactive problem-solving help. Journal of Educational Data Mining , 12(4):24-65, Dec 2020. -
  12. [28][28] Y. Mao, Y. Shi, S. Marwan, T. W. Price, T. Barnes, and M. Chi. Knowing' when' and' where': Temporal-astnn for student learning progression in novice programming tasks. In In: Proceedings of the 14th International Conference on Educational Data Mining (EDM 2021) , 2021. -
  13. [29][29] Y. Mao, R. Zhi, F. Khoshnevisan, T. W. Price, T. Barnes, and M. Chi. One minute is enough: Early prediction of student success and event-level difficulty during a novice programming task. In In: Proceedings of the 12th International Conference on Educational Data Mining (EDM 2019) , 2019. -
  14. [31][31] S. Pandey and G. Karypis. A self-attentive model for knowledge tracing. In In Proceedings of the 12th International Conference on Educational Data Mining (EDM) 2019 , 2019. -
  15. [33][33] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019. -
  16. [34][34] C. Piech, J. Bassen, J. Huang, S. Ganguli, M. Sahami, L. J. Guibas, and J. Sohl-Dickstein. Deep knowledge tracing. Advances in Neural Information Processing Systems , 28, 2015. -
  17. [42][42] Y. Shi, Y. Mao, T. Barnes, M. Chi, and T. W. Price. More with less: Exploring how to use deep learning effectively through semi-supervised learning for automatic bug detection in student code. In In Proceedings of the 14th International Conference on Educational Data Mining (EDM) 2021 , 2021. -
  18. [43][43] Y. Shi, K. Shah, W. Wang, S. Marwan, P. Penmetsa, and T. Price. Toward semi-automatic misconception discovery using code embeddings. In LAK21: 11th International Learning Analytics and Knowledge Conference , pages 606-612, 2021. -
  19. [44][44] D. Shin, Y. Shim, H. Yu, S. Lee, B. Kim, and Y. Choi. Saint+: Integrating temporal features for ednet correctness prediction. In LAK21: 11th International Learning Analytics and Knowledge Conference , pages 490-496, 2021. -
  20. [45][45] Y. Su, Q. Liu, Q. Liu, Z. Huang, Y. Yin, E. Chen, C. Ding, S. Wei, and G. Hu. Exercise-enhanced sequential modeling for student performance prediction. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018. -
  21. [46][46] V. Swamy, A. Guo, S. Lau, W. Wu, M. Wu, Z. Pardos, and D. Culler. Deep knowledge tracing for free-form student code progression. In International Conference on Artificial Intelligence in Education , pages 348-352. Springer, 2018. -
  22. [47][47] M. L. Swartz and M. Yazdani. Intelligent tutoring systems for foreign language learning: The bridge to international communication , volume 80. Springer Science & Business Media, 2012. -
  23. [48][48] N. Truong, P. Roe, and P. Bancroft. Static analysis of students' java programs. In Proceedings of the Sixth Australasian Conference on Computing Education-Volume 30 , pages 317-325. Citeseer, 2004. -
  24. [50][50] K. VanLehn. The Behavior of Tutoring Systems. International Journal of Artificial Intelligence in Education , 16(3):227-265, 2006. -
  25. [51][51] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017. -
  26. [52][52] L. Wang, A. Sy, L. Liu, and C. Piech. Learning to represent student knowledge on programming exercises using deep learning. In In: Proceedings of the 10th International Conference on Educational Data Mining (EDM 2017) , 2017. -
  27. [53][53] S. Wang, A. Ororbia, Z. Wu, K. Williams, C. Liang, B. Pursel, and C. L. Giles. Using prerequisites to extract concept maps from textbooks. In Proceedings of the 25th ACM international conference on Information and Knowledge Management , pages 317-326, 2016. -
  28. [54][54] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning , pages 2048-2057. PMLR, 2015. -
  29. [55][55] M. Yudelson, R. Hosseini, A. Vihavainen, and P. Brusilovsky. Investigating automated student modeling in a java mooc. In In Proceedings of the 7th International Conference on Educational Data Mining (EDM) 2014 , 2014. -
  30. [56][56] M. V. Yudelson, K. R. Koedinger, and G. J. Gordon. Individualized bayesian knowledge tracing models. In International Conference on Artificial Intelligence in Education , pages 171-180. Springer, 2013. -
  31. [57][57] L. Zhang, X. Xiong, S. Zhao, A. Botelho, and N. T. Heffernan. Incorporating rich features into deep knowledge tracing. In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale , pages 169-172, 2017. -
  32. [58][58] G. Zhou, H. Azizsoltani, M. S. Ausin, T. Barnes, and M. Chi. Hierarchical reinforcement learning for pedagogical policy induction. In International conference on Artificial Intelligence in Education , pages 544-556. Springer, 2019.

Cited by

Sorted by citations

Related

Metadata

Authors
Shi, Y., Chi, M., Barnes, T., & Price, T.
ORCID
Venue
Zenodo
Publisher
Zenodo
Published
2022-07-18
DOI
10.5281/zenodo.6853105
OpenAlex
W6912793668
Cited by
7
References
0
Citations by year
2026: 3 · 2025: 1 · 2024: 3
FWCI
0.43
Type
conference paper
Language
en
Open access
Licence
Locations
  • Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
  • Zenodo (CERN European Organization for Nuclear Research) (cc-by)
Funders
National Science Foundation (2013502)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
Keywords
knowledge tracing, deep knowledge tracing, programming education, student performance prediction, code features, attention mechanism, domain-specific features, Bayesian Knowledge Tracing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review