The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.
A Self-Attentive model for Knowledge Tracing
Abstract
Knowledge tracing is the task of modeling each student's mastery of knowledge concepts (KCs) as (s)he engages with a sequence of learning activities. Each student's knowledge is modeled by estimating the performance of the student on the learning activities. It is an important research area for providing a personalized learning platform to students. In recent years, methods based on Recurrent Neural Networks (RNN) such as Deep Knowledge Tracing (DKT) and Dynamic Key-Value Memory Network (DKVMN) outperformed all the traditional methods because of their ability to capture complex representation of human learning. However, these methods face the issue of not generalizing well while dealing with sparse data which is the case with real-world data as students interact with few KCs. In order to address this issue, we develop an approach that identifies the KCs from the student's past activities that are \textit{relevant} to the given KC and predicts his/her mastery based on the relatively few KCs that it picked. Since predictions are made based on relatively few past activities, it handles the data sparsity problem better than the methods based on RNN. For identifying the relevance between the KCs, we propose a self-attention based approach, Self Attentive Knowledge Tracing (SAKT). Extensive experimentation on a variety of real-world dataset shows that our model outperforms the state-of-the-art models for knowledge tracing, improving AUC by 4.43% on average.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Domain-specific features may be just as important as model structure for improving KT performance.
“suggesting that domain-specific features may be just as important as model structure.”
From Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks (2022) · cites this paper as [31]
The Introduction says SAKT uses relevant knowledge concepts from past interactions and reports improved KT performance. But the paper does not compare the importance of domain-specific features with model structure; its ablation study varies architectural components, not that trade-off. No page numbers are available in our copy. (LLM, conf=0.78)
Experimental — accuracy not yet measuredRecent models have outperformed DKT by only about 2-4% AUC, suggesting DKT is still representative of modern deep KT models.
“Some recent models have outperformed DKT, but only by about 2-4% AUC [31, 44], suggesting that DKT is still representative of modern deep KT models.”
From Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks (2022) · cites this paper as [31]
Table 3 (listed under Conclusion and Future Work; no page given) shows SAKT exceeding DKT by about 2–4 AUC percentage points on ASSIST2009 and STATICS. But the gain is much larger on ASSIST2015 (0.854 versus 0.736), so the source does not support treating 2–4% as a general upper bound. The statement that DKT remains representative is the citing authors’ inference. (LLM, conf=0.87)
Experimental — accuracy not yet measuredCode-DKT's improvement is comparable to SAINT over DKT on EdNet (+2.76% AUC) and SAKT over DKT on various datasets (+3.8%).
“This is comparable to the improvement of SAINT over DKT on the EdNet dataset (they achieved +2.76% in AUC), or SAKT over DKT on various datasets (+3.8%) [31].”
From Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks (2022) · cites this paper as [31]
Table 3 reports average AUCs of 0.824 for SAKT and 0.786 for DKT across five datasets, a difference of 0.038 (3.8 percentage points). Our copy does not report a SAINT comparison on EdNet or the claimed +2.76% AUC improvement. (LLM, conf=0.90)
Experimental — accuracy not yet measuredThe improvement of Code-DKT over DKT is comparable to that of other modern KT models over DKT (2-4% AUC).
“This improvement is comparable to that of other modern KT models over DKT (2-4%) [31, 44]”
From Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks (2022) · cites this paper as [31]
AUC is a standard evaluation metric for KT models and is more appropriate than accuracy for imbalanced datasets like ours.
“Our primary performance metric is AUC, a standard evaluation metric for KT models [34, 44, 31], as it uses the predicted probability of success, rather than a binary correctness prediction, and is more appropriate than accuracy for imbalanced datasets like ours (23% positive).”
From Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks (2022) · cites this paper as [31]
DKT is a commonly used baseline in recent KT papers.
“as it is a commonly used baseline in other more recent KT papers [31, 44].”
From Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks (2022) · cites this paper as [31]
DKT has historically performed well on some other small datasets (e.g. ASSIST-Chall and STATICS from [31] with 300-700 students).
“Additionally, DKT has historically performed well on some other small datasets (e.g. the 'ASSIST-Chall' and 'STATICS' datasets from [31] with 300-700 students).”
From Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks (2022) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Flagged first, then by citations
Research works printed but not linked: 10
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint Figure 4: Training Efficiency on ASSIST2009 dataset. arXiv:1607.06450 (2016). · arXiv
- Wang-Cheng Kang and Julian McAuley. 2018. Self-Attentive Sequential Recommendation. CoRR abs/1808.09781 (2018). · arXiv
- Mohammad Khajah, Robert V Lindsey, and Michael C Mozer. 2016. How deep is knowledge tracing? arXiv preprint arXiv:1604.02416 (2016). · arXiv
- Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). · arXiv
- Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge tracing. In Advances in Neural Information Processing Systems. 505-513.
- Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. 2016. One-shot learning with memory-augmented neural networks. arXiv preprint arXiv:1605.06065 (2016). · arXiv
- John Self. 1990. Theoretical foundations for intelligent tutoring systems. Journal of Artificial Intelligence in Education 1, 4 (1990), 3-14.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 5998-6008.
- Chun-Kit Yeung and Dit-Yan Yeung. 2018. Addressing two problems in deep knowledge tracing via prediction-consistent regularization. arXiv preprint arXiv:1806.02180 (2018). · arXiv
- Jiani Zhang, Xingjian Shi, Irwin King, and Dit-Yan Yeung. Dynamic key-value memory networks for knowledge tracing. In Proceedings of the 26th International Conference on World Wide Web.
Cited by
Sorted by citations
AI-based personalized E-learning systems: Issues, challenges, and solutions
Murtaza, M., Ahmed, Y., Shamsi, J. A., Sherwani, F., & Usman, M. (2022) · IEEE Access · doi:10.1109/access.2022.3193938 · Cited by 386
Proactive and reactive engagement of artificial intelligence methods for education: a review
Mallik, S., & Gangopadhyay, A. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1151391 · Cited by 139
Metadata
- Authors
- Pandey, S., & Karypis, G.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- arXiv
- 1907.06837
- References
- 1
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
