Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 34

The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.

Open-ended Knowledge Tracing for Computer Science Education

Liu, Naiming, Wang, Zichao, Baraniuk, Richard, Lan, Andrew·2022·Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2022.emnlp-main.254

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Open-ended knowledge tracing (OKT) block diagram. We update the student's current knowledge state h t +1 using the last question p t and actual student code x t . We then combine it with the next question statement p t +1 to generate our prediction of the actual student code ˆ x t +1 .
Figure 1 Open-ended knowledge tracing (OKT) block diagram. We update the student's current knowledge state h t +1 using the last question p t and actual student code x t . We then combine it with the next question statement p t +1 to generate our prediction of the actual student code ˆ x t +1 .CC-BY · From the PDF
Table 3: OKT generated code vs. actual student code for two questions (differences highlighted in red boxes).
Table 3: OKT generated code vs. actual student code for two questions (differences highlighted in red boxes).CC-BY · From the PDF
Image 3
CC-BY · From the PDF
Figure 3 Comparison of the knowledge state spaces learned by DKT (left), DKT with code embeddings as input (Mao et al., 2021) (middle), and OKT (right). OKT learns a knowledge space with distinct clusters that capture variations in actual student code.
Figure 3 Comparison of the knowledge state spaces learned by DKT (left), DKT with code embeddings as input (Mao et al., 2021) (middle), and OKT (right). OKT learns a knowledge space with distinct clusters that capture variations in actual student code.CC-BY · From the PDF
Figure 4 Visualization of actual student code (blue) compared to predicted code (green) for a new question unseen during training. Code pairs that are close in the code embedding space are connected (red).
Figure 4 Visualization of actual student code (blue) compared to predicted code (green) for a new question unseen during training. Code pairs that are close in the code embedding space are connected (red).CC-BY · From the PDF
Figure 5 Visualization of CodeBLEU metric versus number of student responses (left), rate of correct submissions (middle) and Dist-1 metric (right) in each question. Each point represents one question.
Figure 5 Visualization of CodeBLEU metric versus number of student responses (left), rate of correct submissions (middle) and Dist-1 metric (right) in each question. Each point represents one question.CC-BY · From the PDF
Figure 6 Four sample submissions of two students corresponding to the top right figures, respectively. One student gradually proceeded to a correct code while the other got stuck.
Figure 6 Four sample submissions of two students corresponding to the top right figures, respectively. One student gradually proceeded to a correct code while the other got stuck.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 10 references

Research works printed but not linked: 41

  1. Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019. code2vec: Learning distributed rep- resentations of code. Proc. ACM Program. Lang., 3(POPL):1-29.
  2. John R Anderson and Robin Jeffries. 1985. Novice lisp errors: Undetected losses of information from work- ing memory. Human-Computer Interact., 1(2):107- 131.
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. · arXiv
  4. John Seely Brown and Richard R Burton. 1978. Diag- nostic models for procedural bugs in basic mathemat- ical skills. Cogn. sci., 2(2):155-192.
  5. Benoît Choffin, Fabrice Popineau, Yolaine Bourda, and Jill-Jênn Vie. 2019. DAS3H: Modeling student learning and forgetting for pptimally scheduling dis- tributed practice of skills. In Proc. Int. Conf. Educ. Data Mining, pages 29-38.
  6. Youngduck Choi, Youngnam Lee, Dongmin Shin, Junghyun Cho, Seoyon Park, Seewoo Lee, Jineon Baek, Chan Bae, Byungsoo Kim, and Jaewe Heo. 2020. Ednet: A large-scale hierarchical dataset in education. In Int. Conf. Artif. Intell. Educ., pages 69-73. Springer.
  7. Aditya Desai, Sumit Gulwani, Vineet Hingorani, Nidhi Jain, Amey Karkare, Mark Marron, and Subhajit Roy. 2016. Program synthesis using natural language. In Proc. 38th Int. Conf. Softw. Eng., pages 345-356.
  8. Molly Q Feldman, Ji Yong Cho, Monica Ong, Sumit Gulwani, Zoran Popović, and Erik Andersen. 2018. Automatic diagnosis of students' misconceptions in K-8 mathematics. In Proc. CHI Conf. Human Fac- tors Comput. Syst., pages 1-12.
  9. Junchen Feng, Bo Zhang, Yuchen Li, and Qiushi Xu. 2019. Bayesian diagnosis tracing: Application of procedural misconceptions in knowledge tracing. In Proc. Int. Conf. Artif. Intell. Educ., pages 84-88. Springer.
  10. Aritra Ghosh, Neil Heffernan, and Andrew S Lan. 2020. Context-aware attentive knowledge tracing. In Proc. ACM SIGKDD, pages 2330-2339.
  11. Aritra Ghosh, Jay Raspat, and Andrew Lan. 2021. Op- tion tracing: Beyond correctness analysis in knowl- edge tracing. In Int. Conf. Artif. Intell. Educ., pages 137-149. Springer.
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735- 1780.
  13. MM Khajah, Y Huang, JP González-Brenes, MC Mozer, and P Brusilovsky. 2014. Integrating knowledge trac- ing and item response theory: A tale of two frame- works. In Proc. Int. Workshop Personalization Ap- proaches Learn. Environ., volume 1181, pages 7-15.
  14. Alexander LeClair and Collin McMillan. 2019. Recom- mendations for datasets for source code summariza- tion. arXiv preprint arXiv:1904.02660. · arXiv
  15. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objec- tive function for neural conversation models. In Proc. Conf. North Amer. Chapter Assoc. Comput. Linguis- tics Human Lang. Technol., pages 110-119.
  16. Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. 2019. Ekt: Exercise- aware knowledge tracing for student performance pre- diction. IEEE Trans. Knowl. Data Eng., 33(1):100- 115.
  17. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664. · arXiv
  18. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. In Proc. NeurIPS, pages 3111-3119.
  19. Shalini Pandey and Jaideep Srivastava. 2020. Rkt: Relation-aware self-attention for knowledge tracing. arXiv preprint arXiv:2008.12736. · arXiv
  20. Zach A Pardos and Neil T Heffernan. 2010. Modeling individualization in a Bayesian networks implemen- tation of knowledge tracing. In Proc. Int. Conf. User Model. Adaptation Personalization, pages 255-266.
  21. Philip Pavlik Jr, Hao Cen, and Kenneth Koedinger. 2009. Performance factors analysis-A new alternative to knowledge tracing. In Proc. Int. Conf. Artif. Intell. Educ.
  22. Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. 2015a. Deep knowledge trac- ing. In Proc. NeurIPS, pages 505-513.
  23. Chris Piech, Mehran Sahami, Jonathan Huang, and Leonidas Guibas. 2015b. Autonomously generat- ing hints by inferring problem solving policies. In Proc. ACM conf. learn. Scale, pages 195-204.
  24. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  25. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. arXiv preprint arXiv:2009.10297. · arXiv
  26. Steven Ritter, John R Anderson, Kenneth R Koedinger, and Albert Corbett. 2007. Cognitive tutor: Applied research in mathematics education. Psychonomic Bulletin & Review, 14(2):249-255.
  27. Kelly Rivers and Kenneth R Koedinger. 2017. Data- driven hint generation in vast solution spaces: a self- improving python programming tutor. Int. J. Artif. Intell. Educ., 27(1):37-64.
  28. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical rea- soning abilities of neural models. arXiv preprint arXiv:1904.01557. · arXiv
  29. Dongmin Shin, Yugeun Shim, Hangyeol Yu, Seewoo Lee, Byungsoo Kim, and Youngduck Choi. 2021. Saint+: Integrating temporal features for ednet cor- rectness prediction. In 11th Int. Learn. Analytics Knowl. Conf., pages 490-496.
  30. Adish Singla and Nikitas Theodoropoulos. 2022. From {Solution Synthesis} to {Student Attempt Synthesis} for block-based visual programming tasks. arXiv preprint arXiv:2205.01265. · arXiv
  31. John P Smith III, Andrea A DiSessa, and Jeremy Roschelle. 1994. Misconceptions reconceived: A constructivist analysis of knowledge in transition. j. learn. sci., 3(2):115-163.
  32. W. J. van der Linden and R. K. Hambleton. 2013. Hand- book of Modern Item Response Theory. Springer Science & Business Media.
  33. Laurens van der Maaten and Geoffrey Hinton. 2008. Vi- sualizing data using t-sne. JMLR, 9(86):2579-2605.
  34. Kurt VanLehn. 1982. Bugs are not enough: Empirical studies of bugs, impasses and repairs in procedural skills. J. Math. Behavior.
  35. Tianqi Wang, Fenglong Ma, Yaqing Wang, Tang Tang, Longfei Zhang, and Jing Gao. 2021. Towards learn- ing outcome prediction via modeling question expla- nations and student responses. In Proc. SIAM Int. Conf. Data Mining, pages 693-701. SIAM.
  36. Yutao Wang and Neil Heffernan. 2013. Extending knowledge tracing to allow partial credit: Using con- tinuous versus binary nodes. In Int. conf. artif. intell. educ., pages 181-188. Springer.
  37. Yang Yang, Jian Shen, Yanru Qu, Yunfei Liu, Kerong Wang, Yaoming Zhu, Weinan Zhang, and Yong Yu. 2020. Gikt: A graph-based interaction model for knowledge tracing. In Proc. Joint Eur. Conf. Mach. Learn. Knowl. Discovery Databases.
  38. Michael V Yudelson, Kenneth R Koedinger, and Geof- frey J Gordon. 2013. Individualized bayesian knowl- edge tracing models. In Int. Conf. artif. intell. educ., pages 171-180. Springer.
  39. Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. 2019. A novel neural source code representation based on abstract syntax tree. In IEEE/ACM 41st Int. Conf. Software Eng., pages 783-794. IEEE.
  40. Jiani Zhang, Xingjian Shi, Irwin King, and Dit-Yan Yeung. 2017. Dynamic key-value memory networks for knowledge tracing. In Proc. Int. Conf. World Wide Web, pages 765-774.
  41. Renyu Zhu, Dongxiang Zhang, Chengcheng Han, Ming Gao, Xuesong Lu, Weining Qian, and Aoying Zhou. 2021. Programming knowledge tracing: A compre- hensive dataset and a new model. arXiv preprint arXiv:2112.08273. · arXiv

Cited by

See all 34

Metadata

Authors
Liu, N., Wang, Z., Baraniuk, R., & Lan, A.
Venue
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Published
2022
Pages
3849-3862
Cited by
34
References
10

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Open-ended Knowledge Tracing for Computer Science Education. https://app.cochat.ai/papers/10.18653/v1/2022.emnlp-main.254
The paper
Liu, N., Wang, Z., Baraniuk, R., & Lan, A. (2022). Open-ended Knowledge Tracing for Computer Science Education. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2022.emnlp-main.254

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.