New Methods for Confusion Detection in Course Forums: Student, Teacher, and Machine
Abstract
This article provides computational and rule-based approaches for detecting confusion that is expressed in students' comments in couse forums. To obtain reliable, ground truth data about which posts exhibit student confusion, we designed a decision tree that facilitates the manual labeling of forum posts by experts. However, manual labeling is costly in time and resources, which limits the amount of data that can be generated using this process. Our strategy for overcoming these limitations was to generate rules for detecting confusion based on student input via hashtags, which reflect the student's affective states. We show that the resulting rules closely align with the ground truth judgement of experts. We next applied these rules to datasets of students' forum posts in a large-scale biology course, thereby automatically generating thousands of labeled instances of “confused posts.” Finally, the resulting dataset was used to train a machine learning model for detecting whether students' posts exhibit confusion in the absence of hashtags. In this task, the pretrained language model based on bidirectional encoder representation from transformers (BERT) was able to outperform traditional machine learning models for classifying confusion in posts. This model was also able to generalize and detect student confusion across different offerings of the same course. Ultimately, the use of pretrained language models of this type will provide teachers with better technologies for detecting and alleviating confusion in online discussion forums by leveraging the combined input of teachers and students.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
F1 scores for detecting student confusion in the course forum are around 0.77.
“the F1 scores for detecting student confusion in the course forum (Geller et al., 2021) and students' off-task messages in game-based collaborative learning (Carpenter et al., 2020) are around 0.77 and 0.67, respectively.”
From Practical and ethical challenges of large language models in education: A systematic scoping review (2024) · cites this paper as Geller et al., 2021
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Term-Weighting Approaches in Automatic Text Retrieval
Salton, G., & Buckley, C. (1988) · Information Processing & Management · doi:10.1016/0306-4573(88)90021-0 · Cited by 11,421 · see citing works
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559 · see citing works
Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002) · Educational Psychologist · doi:10.1207/s15326985ep3702_4 · Cited by 3,931 · see citing works
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
The Psychology Of Intelligence
Piaget, J. (2005) · Routledge eBooks · doi:10.4324/9780203981528 · Cited by 3,029 · see citing works
Linguistic Inquiry and Word Count (LIWC)
Chung, C. K., & Pennebaker, J. W. (2012) · IGI Global eBooks · doi:10.4018/978-1-60960-741-8.ch012 · Cited by 2,152 · see citing works
Does discovery-based instruction enhance learning?
Alfieri, L., Brooks, P. J., Aldrich, N. J., & Tenenbaum, H. R. (2011) · Journal of Educational Psychology · doi:10.1037/a0021017 · Cited by 1,338 · see citing works
Dynamics of affective states during complex learning
D’Mello, S. K., & Graesser, A. C. (2011) · Learning and Instruction · doi:10.1016/j.learninstruc.2011.10.001 · Cited by 1,101 · see citing works
Semi-supervised Sequence Learning
Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994 · see citing works
Kapur, M. (2008) · Cognition and Instruction · doi:10.1080/07370000802212669 · Cited by 867 · see citing works
Confusion can be beneficial for learning.
D'Mello, S. K., Lehman, B. A., Pekrun, R., & Graesser, A. (2014) · Learning and Instruction · doi:10.1016/j.learninstruc.2012.05.003 · Cited by 782 · see citing works
(2011) · Cited by 725 · see citing works
Enhanced Sentiment Learning Using Twitter Hashtags and Smileys
Davidov, D., Tsur, O., & Rappoport, A. (2010) · Cited by 698 · see citing works
Michelene, T. H. (2009) · doi:10.4324/9780203874813-9 · Cited by 531 · see citing works
The Reliability, Validity, and Utility of Self-Assessment
Ross, J. A. (2020) · doi:10.7275/9wph-vv65 · Cited by 454 · see citing works
Comparative experiments on learning information extractors for proteins and their interactions
Bunescu, R., Ruifang Ge, A., Kate, R. J., Marcotte, E. M., Mooney, R. J., Ramani, A. K., & Wong, Y. W. (2004) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2004.07.016 · Cited by 428 · see citing works
Using Hashtags to Capture Fine Emotion Categories from Tweets
Mohammad, S. M., & Kiritchenko, S. (2014) · Computational Intelligence · doi:10.1111/coin.12024 · Cited by 409 · see citing works
Revisiting readability: a unified framework for predicting text quality
Pitler, E., & Nenkova, A. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613742 · Cited by 375 · see citing works
Understanding Difficulties and Resulting Confusion in Learning: An Integrative Review
Lodge, J. M., Kennedy, G. E., Lockyer, L., Arguel, A., & Pachman, M. (2018) · Frontiers in Education · doi:10.3389/feduc.2018.00049 · Cited by 322 · see citing works
Learning the structure of Markov logic networks
Kok, S., & Domingos, P. (2005) · International Conference on Machine Learning · doi:10.1145/1102351.1102407 · Cited by 301 · see citing works
emoji2vec: Learning Emoji Representations from their Description
Eisner, B., Rocktäschel, T., Augenstein, I., Bošnjak, M., & Riedel, S. (2016) · International Workshop on Natural Language Processing for Social Media (SocialNLP) · doi:10.18653/v1/w16-6208 · Cited by 291 · see citing works
Smart schools : better thinking and learning for every child
Perkins, D. N. (1995) · Medical Entomology and Zoology · Cited by 288 · see citing works
Toward a Theory of Impasse-Driven Learning
VanLehn, K. A. (1988) · Springer eBooks · doi:10.1007/978-1-4684-6350-7_2 · Cited by 206 · see citing works
Wheel-Spinning: Students Who Fail to Master a Skill
Beck, J. E., & Gong, Y. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39112-5_44 · Cited by 162 · see citing works
Graesser, A. C., Lu, S., Olde, B. A., Cooper, E., & Whitten, S. N. (2005) · Memory & Cognition · doi:10.3758/bf03193225 · Cited by 150 · see citing works
Exploring the Effect of Confusion in Discussion Forums of Massive Open Online Courses
Yang, D., Wen, M.-M., Howley, I., Kraut, R. E., & Rosé, C. (2015) · ACM Conference on Learning @ Scale · doi:10.1145/2724660.2724677 · Cited by 137 · see citing works
Almatrafi, O., Johri, A., & Rangwala, H. (2017) · Computers & Education · doi:10.1016/j.compedu.2017.11.002 · Cited by 134 · see citing works
Shute, V. J., D’Mello, S. K., Baker, R. S., Cho, K., Bosch, N., Ocumpaugh, J. L., Ventura, M., & Almeda, M. V. (2015) · Computers & Education · doi:10.1016/j.compedu.2015.08.001 · Cited by 129 · see citing works
YouEDU: Addressing Confusion in MOOC Discussion Forums by Recommending Instructional Video Clips.
Agrawal, A., Venkatraman, J., Leonard, S., & Paepcke, A. (2015) · Educational Data Mining · Cited by 107 · see citing works
Successful classroom deployment of a social document annotation system
Zyto, S., Karger, D., Ackerman, M., & Mahajan, S. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208326 · Cited by 90 · see citing works
Self-citationTowards triggering higher-order thinking behaviors in MOOCs
Wang, X., Wen, M., & Rosé, C. P. (2016) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2883851.2883964 · Cited by 89 · see citing works
A Convolution-LSTM-Based Deep Neural Network for Cross-Domain MOOC Forum Post Classification
Wei, X., Lin, H., Yang, L., & Yu, Y. (2017) · Information · doi:10.3390/info8030092 · Cited by 81 · see citing works
Arguel, A., Lockyer, L., Lipp, O., Lodge, J., & Kennedy, G. (2017) · Journal of Educational Computing Research · doi:10.1177/0735633116674732 · Cited by 62 · see citing works
Miller, K., Zyto, S., Karger, D. R., Yoo, J., & Mazur, E. (2016) · Physical Review Physics Education Research · doi:10.1103/physrevphyseducres.12.020143 · Cited by 58 · see citing works
Self-citationEduBERT: Pretrained Deep Language Models for Learning AnalyticsPreprint
Clavié, B., & Gal, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.00690 · Cited by 37 · see citing works
Self-citationMarkov Networks for Detecting Overalpping Elements in Sequence Data
Craven, M. W., & Bockhorst, J. P. (2004) · neural information processing systems · Cited by 36 · see citing works
Copying@Scale: Using Harvesting Accounts for Collecting Correct Answers in a MOOC
Alexandron, G., Ruipérez-Valiente, J. A., Chen, Z.-Z., Muñoz-Merino, P., & Pritchard, D. E. (2017) · Comput. Educ. · doi:10.1016/j.compedu.2017.01.015 · Cited by 36 · see citing works
Plaut, S. (2006) · Curriculum Inquiry · doi:10.1111/j.1467-873x.2006.00364.x · Cited by 32 · see citing works
Using Student Annotated Hashtags and Emojis to Collect Nuanced Affective States
Zhang, A. X., Igo, M. M., Facciotti, M. T., & Karger, D. R. (2017) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3051457.3054014 · Cited by 22 · see citing works
Self-citationZeng, Z., Chaturvedi, S., & Bhat, S. (2017) · Educational Data Mining · Cited by 20 · see citing works
The Relationships Between Sequences of Affective States and Learner Achievement
Rodrigo, M. M. T., BAKER, R., & Nabos, J. Q. (2010) · International Conference on Computers in Education · doi:10.58459/icce.2010.38 · Cited by 19 · see citing works
Classifying and visualizing students' cognitive engagement in course readings
Yogev, E., Gal, K., Karger, D. R., Facciotti, M. T., & Igo, M. M. (2018) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3231644.3231648 · Cited by 15 · see citing works
Self-citationGeller, S. A., Hoernle, N., Gal, K., Segal, A., Zhang, A. X., Karger, D. R., Facciotti, M. T., & Igo, M. M. (2020) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3375462.3375485 · Cited by 13 · see citing works
Self-citationAdapting Feedback Types According to Students’ Affective States
Grawemeyer, B., Mavrikis, M., Holmes, W., & Gutiérrez-Santos, S. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-19773-9_68 · Cited by 12 · see citing works
Flagged first, then by citations
2 of the 49 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Practical and ethical challenges of large language models in education: A systematic scoping review
Yan, L., Sha, L., Zhao, L., Li, Y., Martinez‐Maldonado, R., Chen, G., Li, X., Jin, Y., & Gašević, D. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13370 · Cited by 935 · see citing works
“For example, the F1 scores for detecting student confusion in the course forum (Geller et al., 2021) and students’ off-task messages in game-based collaborative learning (Carpenter et al.”
BackgroundInfluentialOptimizing Formative Assessment with Learning Analytics
Banihashem, S. K., Gašević, D., Noroozi, O., Jarodzka, H., Brinke, D. J.-T., & Drachsler, H. (2025) · Review of Educational Research · doi:10.3102/00346543251370753 · Cited by 35 · see citing works
Security and Privacy of Large Language Models: Threat Taxonomy, Ethical Implications, and Governance
Pribisalić, M., & Martinčić-Ipšić, S. (2026) · AI · doi:10.3390/ai7050152 · Cited by 9 · see citing works
Unveiling Uncertainty: Supporting Learners Through NLP-Driven Confusion Identification
Jhajj, G., Dewan, M. A. A., & Lin, F. O. (2023) · IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech) · doi:10.1109/dasc/picom/cbdcom/cy59711.2023.10361304 · Cited by 4 · see citing works
Student Perceptions Influenced by Remote Laboratory Infrastructure
Reid, D. P., & Drysdale, T. D. (2023) · Experiment@International Conference · doi:10.1109/exp.at2358782.2023.10545902 · Cited by 2 · see citing works
Ebinç, S., Kalkan, Z., Oruç, Z., Sezgin, Y., Urakçı, Z., Küçüköner, M., Ma, K., & Işıkdoğan, A. (2023) · TURKDERM · doi:10.4274/turkderm.galenos.2023.52721 · Cited by 1 · see citing works
core technology behind and beyond ChatGPT
Leong, K., Sung, A., & Jones, L. (2023) · IJERI International Journal of Educational Research and Innovation · doi:10.46661/ijeri.8449 · Cited by 1 · see citing works
Investigating the Understandability of Review Comments on Code Change Requests
Rahman, M. S., Codabux, Z., & Roy, C. K. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00087
Pinto, A. D. S., Longhi, M. T., Behar, P. A., & Akazaki, J. M. (2026) · Smart innovation, systems and technologies · doi:10.1007/978-3-032-20458-5_8
- Authors
- Geller, S. A., Gal, K., Segal, A., Sripathi, K., Kim, H. G., Facciotti, M. T., Igo, M., Hoernle, N., & Karger, D.
- Affiliations
- Ben-Gurion University of the Negev (IL)
- University of California, Davis (US)
- University of Edinburgh (GB)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Venue
- IEEE Transactions on Learning Technologies
- Published
- 2021-10-01
- Volume
- 14
- Issue
- 5
- Pages
- 665-679
- ISSN
- 1939-1382 (ISSN-L)
- 2372-0050
- OpenAlex
- W3209636133
- Cited by
- 9
- References
- 49
- Citations by year
- 2026: 2 · 2025: 2 · 2023: 5
- FWCI
- 1.06
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Learning Technologies (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- National Science Foundation (1915724)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- online forums, hashtags, affective states, rule-based detection, BERT, pre-trained language models, automatic annotation, biology course, online learning analytics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
