Automatic Detection of Learning-Centered Affective States in the Wild
Abstract
Affect detection is a key component in developing intelligent educational interfaces that are capable of responding to the affective needs of students. In this paper, computer vision and machine learning techniques were used to detect students' affect as they used an educational game designed to teach fundamental principles of Newtonian physics. Data were collected in the real-world environment of a school computer lab, which provides unique challenges for detection of affect from facial expressions (primary channel) and gross body movements (secondary channel) - up to thirty students at a time participated in the class, moving around, gesturing, and talking to each other. Results were cross validated at the student level to ensure generalization to new students. Classification was successful at levels above chance for off-task behavior (area under receiver operating characteristic curve or (AUC = .816) and each affective state including boredom (AUC =.610), confusion (.649), delight (.867), engagement (.679), and frustration (.631) as well as a five-way overall classification of affect (.655), despite the noisy nature of the data. Implications and prospects for affect-sensitive interfaces for educational software in classroom environments are discussed.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Methods for recognizing other affective states have been proposed, namely cognitive states like boredom, confusion, delight, concentration and frustration, positive and negative affect from groups of people, or liking/not-liking of online media for predicting buying behaviour for marketing purposes.
“Methods for recognizing other affective states have also been proposed, namely cognitive states like boredom, confusion, delight, concentration and frustration [266], positive and negative affect from groups of people [267] or liking/not-linking of online media for predicting buying behaviour for marketing purposes [268].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [266]
Most approaches target primitive facial expressions of affect, but others recognize cognitive states, positive and negative affect from groups of people, and liking/not-liking of online media for predicting buying behaviour for marketing purposes.
“Most approaches target primitive FEs of affect, but others recognize cognitive states [266], postive and negative affect from groups of people [267] and liking/not-linking of online media for predicting buying behaviour for marketing purposes [268].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [266]
In naturalistic environments, most approaches target primitive facial expressions of affect, but methods for recognizing cognitive states, positive and negative affect from groups of people, or liking/not-liking of online media for predicting buying behaviour for marketing purposes are also common.
“In naturalistic environments, most approaches target primitive FEs of affect. Methods for recognizing cognitive states [266], positive and negative affect from groups of people [267] or liking/not-linking of online media for predicting buying behaviour for marketing purposes [268] are also common.”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [266]
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · see citing works
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879 · see citing works
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019 · see citing works
Estimating attributes: Analysis and extensions of RELIEF
Kononenko, I. (1994) · Lecture notes in computer science · doi:10.1007/3-540-57868-4_57 · Cited by 3,062 · see citing works
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797 · see citing works
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 4× · see citing works
“Affect detection based on facial features has been the subject of considerable previous research (see [6,42] for reviews); albeit outside of learning contexts and mainly in laboratory settings (see exceptions discussed below).”
BackgroundInfluentialSelf-citationAllison, P. D. (1998) · Cited by 1,721 · see citing works
Dynamics of affective states during complex learning
D’Mello, S. K., & Graesser, A. C. (2011) · Learning and Instruction · doi:10.1016/j.learninstruc.2011.10.001 · Cited by 1,101 · see citing works
Self-citationWEKA: a machine learning workbench
Holmes, G., Donkin, A., & Witten, I. H. (2002) · Australian and New Zealand Intelligent Information Systems Conference (ANZIIS) · doi:10.1109/anziis.1994.396988 · Cited by 939 · see citing works
Facial signs of emotional experience.
Ekman, P., Freisen, W. V., & Ancoli, S. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077722 · Cited by 888 · see citing works
Facing Imbalanced Data--Recommendations for the Use of Performance Metrics
Jeni, L. A., Cohn, J. F., & de la Torre, F. N. (2013) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2013.47 · Cited by 828 · see citing works
Proceedings of the 33rd Annual Conference of the Cognitive Science Society
Dziemianko, M., Keller, F., & Coco, M. I. (2011) · Cited by 730 · see citing works
The Faces of Engagement: Automatic Recognition of Student Engagementfrom Facial Expressions
Whitehill, J., Serpell, Z. N., Lin, Y.-C., Foster, A., & Movellan, J. R. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2316163 · Cited by 694 · see citing works
Proceedings of the 7th International Conference on Educational Data Mining.
Stamper, J. C., Pardos, Z. A., Mavrikis, M., & McLaren, B. M. (2014) · Educational Data Mining · Cited by 667 · see citing works
Affect and learning: An exploratory look into the role of affect in learning with AutoTutor
Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004) · Journal of Educational Media · doi:10.1080/1358165042000283101 · Cited by 630 · see citing works
Automatic prediction of frustration
Kapoor, A., Burleson, W., & Picard, R. W. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.02.003 · Cited by 592 · see citing works
The computer expression recognition toolbox (CERT)
Littlewort, G. C., Whitehill, J., Wu, T., Fasel, I., Frank, M. G., Movellan, J. R., & Bartlett, M. S. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771414 · Cited by 550 · see citing works
Frenzel, A. C., Pekrun, R. H., & Goetz, T. (2007) · Learning and Instruction · doi:10.1016/j.learninstruc.2007.09.001 · Cited by 486 · see citing works
(2007) · Elsevier eBooks · doi:10.1016/b978-0-12-372545-5.x5000-x · Cited by 417 · see citing works
Ivon, A., Cooper David, G., Winslow, B., Park, W. B., Kasia, M., & Robert, C. (2009) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/978-1-60750-028-5-17 · Cited by 382 · see citing works
Multimodal affect recognition in learning environments
Kapoor, A., & Picard, R. W. (2005) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1101149.1101300 · Cited by 342 · see citing works
Journal of Artificial Intelligence Research (JAIR)
Minton, S. (1993) · Cited by 326 · see citing works
Automated Posture Analysis for Detecting Learner's Interest Level
Mota, S., & Picard, R. W. (2003) · IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2003.10047 · Cited by 315 · see citing works
Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors.
Lepper, M. R., Woolverton, M., Mumme, D. L., & Gurtner, J. (1993) · Cited by 293 · see citing works
D'Mello, S. K. (2013) · Journal of Educational Psychology · doi:10.1037/a0032674 · Cited by 290 · see citing works
Self-citationProceedings of the 29th Annual Cognitive Science Society
McNamara, D. S., & Trafton, J. G. (2007) · Cited by 268 · see citing works
McDuff, D., el Kaliouby, R., Senechal, T., Amr, M., Cohn, J. F., & Picard, R. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2013.130 · Cited by 234 · see citing works
The half-life of cognitive-affective states during complex learning
D’Mello, S. K., & Graesser, A. C. (2011) · Cognition & Emotion · doi:10.1080/02699931.2011.613668 · Cited by 196 · see citing works
Self-citationCoherence between Emotion and Facial Expression: Evidence from Laboratory Experiments
Reisenzein, R., Studtmann, M., & Horstmann, G. (2013) · Emotion Review · doi:10.1177/1754073912457228 · Cited by 196 · see citing works
Emotion recognition in the wild challenge 2013
Dhall, A., Goecke, R., Joshi, J., Wagner, M., & Gedeon, T. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531739 · Cited by 178 · see citing works
Exploring Temporal Patterns in Classifying Frustrated and Delighted Smiles
Hoque, M. E., McDuff, D., & Picard, R. W. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.11 · Cited by 164 · see citing works
Towards Sensor-Free Affect Detection in Cognitive Tutor Algebra.
Baker, R. S., Gowda, S. M., Wixon, M., Kalka, J., Wagner, A. Z., Salvi, A., Aleven, V., Kusbit, G. W., Ocumpaugh, J. L., & Rossi, L. M. (2012) · Cited by 154 · see citing works
Self-citationProceedings of the 33rd Annual Conference of the Cognitive Science Society
Dalzel-Job, S., & Oberlander, J. (2011) · Cited by 145 · see citing works
(1994) · Lecture notes in computer science · doi:10.1007/3-540-57868-4 · Cited by 136 · see citing works
Facial Features for Affective State Detection in Learning Environments
McDaniel, B., D’Mello, S. K., King, B. G., Chipman, P., Tapp, K. M., & Graesser, A. C. (2007) · eScholarship (California Digital Library) · Cited by 133 · see citing works
Self-citationAssessment and Learning of Qualitative Physics in Newton’s Playground
Ventura, M., Shute, V. J., & Kim, Y. J. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39112-5_63 · Cited by 110 · see citing works
Self-citationKnowledge Elicitation Methods for Affect Modelling in Education
Porayska-Pomsta, K., Mavrikis, M., D'Mello, S. K., Conati, C., & Baker, R. (2013) · International Journal of Artificial Intelligence in Education · doi:10.3233/jai-130032 · Cited by 100 · see citing works
Self-citationHernandez, J., Hoque, M. E., Drevo, W., & Picard, R. W. (2012) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2370216.2370264 · Cited by 98 · see citing works
Automatically Recognizing Facial Indicators of Frustration: A Learning-centric Analysis
Grafsgaard, J. F., Wiggins, J. B., Boyer, K., Wiebe, E., & Lester, J. C. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.33 · Cited by 95 · see citing works
The Dynamics of Affective Transitions in Simulation Problem-Solving Environments
Baker, R. S., Rodrigo, M. M. T., & Xolocotzin, U. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_58 · Cited by 74 · see citing works
Self-citationBosch, N., Chen, Y., & D’Mello, S. K. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07221-0_5 · Cited by 69 · see citing works
Self-citationAutomatic measurement of ad preferences from facial responses gathered over the Internet
McDuff, D., El Kaliouby, R., Senechal, T., Demirdjian, D., & Picard, R. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2014.01.004 · Cited by 52 · cited 1× · see citing works
“This dataset has been used for detection of whether viewers liked the commercials (AUC = .82) and wanted to view them again (AUC = .79) [32].”
MethodImpact of action unit detection in automatic emotion recognition
Sénéchal, T., Bailly, K., & Prévost, L. (2012) · Pattern Analysis and Applications · doi:10.1007/s10044-012-0279-5 · Cited by 20 · see citing works
Ocumpaugh, J. L., Baker, R. S., Kamarainen, A. M., & Metcalf, S. (2014) · International Conference on User Modeling, Adaptation, and Personalization · Cited by 6 · see citing works
Self-citationDynamical Emotions: Bodily Dynamics of Affect during Problem Solving
D’Mello, S. K. (2011) · eScholarship (California Digital Library) · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527 · see citing works
Stephanidis, C., Salvendy, G., Antona, M., Chen, J. Y. C., Dong, J., Duffy, V. G., Fang, X., Fidopiastis, C., Fragomeni, G., Fu, L. P., et al. (2019) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2019.1619259 · Cited by 524 · see citing works
Engagement detection in online learning: a review
Dewan, M. A. A., Murshed, M., & Lin, F. O. (2019) · Smart Learning Environments · doi:10.1186/s40561-018-0080-z · Cited by 353 · see citing works
Computer vision for assistive technologies
Leo, M., Medioni, G. G., Trivedi, M. M., Takeo Kanade, A., & Farinella, G. M. (2016) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2016.09.001 · Cited by 228 · see citing works
Multimodal data as a means to understand the learning experience
Giannakos, M. N., Sharma, K., Pappas, I. O., Kostakos, V., & Velloso, E. (2019) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.02.003 · Cited by 210 · see citing works
Automatic Recognition of Student Engagement Using Deep Learning and Facial Expression
Nezami, O. M., Dras, M., Hamey, L., Richards, D., Wan, S., & Paris, C. L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-46133-1_17 · Cited by 138 · see citing works
Facial Emotion Recognition of Students using Convolutional Neural Network
Lasri, I., Solh, A. R., & Elbelkacemi, M. (2019) · International Conference on Intelligent Computing in Data Sciences (ICDS) · doi:10.1109/icds47004.2019.8942386 · Cited by 126 · see citing works
Shute, V. J., D’Mello, S. K., Baker, R. S., Cho, K., Bosch, N., Ocumpaugh, J. L., Ventura, M., & Almeda, M. V. (2015) · Computers & Education · doi:10.1016/j.compedu.2015.08.001 · Cited by 129 · see citing works
Self-citationDeep facial spatiotemporal network for engagement prediction in online learning
Liao, J., Liang, Y., & Pan, J. (2021) · Applied Intelligence · doi:10.1007/s10489-020-02139-8 · Cited by 123 · see citing works
Using Video to Automatically Detect Learner Affect in Computer-Enabled Classrooms
Bosch, N., D'Mello, S. K., Ocumpaugh, J. L., Baker, R., & Shute, V. (2016) · TIIS · doi:10.1145/2946837 · Cited by 124 · see citing works
Self-citationAutomated gaze-based mind wandering detection during computerized learning in classrooms
Hutt, S. J., Krasich, K., Mills, C., Bosch, N., White, S., Brockmole, J. R., & D’Mello, S. K. (2019) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-019-09228-5 · Cited by 109 · see citing works
Self-citationDetecting and Addressing Frustration in a Serious Game for Military Training
DeFalco, J. A., Rowe, J. P., Paquette, L., Georgoulas-Sherry, V., Brawner, K. W., Mott, B. W., Baker, R., & Lester, J. C. (2017) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-017-0152-1 · Cited by 106 · see citing works
Self-citationDonker, M. H., van Gog, T., & Mainhard, T. (2018) · Frontline Learning Research · doi:10.14786/flr.v6i3.372 · Cited by 105 · see citing works
Mehta, N. K., Prasad, S. S., Saurav, S., Saini, R., & Singh, S. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03200-4 · Cited by 97 · see citing works
Profiling sympathetic arousal in a physics course: How active are students?
Pijeira‐Díaz, H. J., Drachsler, H., Kirschner, P. A., & Järvelä, S. (2018) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12271 · Cited by 91 · see citing works
Behera, A., Matthew, P. W., Keidel, A., Vangorp, P., Fang, H., & Canning, S. (2020) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-020-00195-2 · Cited by 85 · see citing works
Task Engagement as Personalization Feedback for Socially-Assistive Robots and Cognitive Training
Tsiakas, K., Abujelala, M., & Makedon, F. S. (2018) · Technologies · doi:10.3390/technologies6020049 · Cited by 80 · see citing works
An affective and Web 3.0-based learning environment for a programming language
Cabada, R. Z., Estrada, M. L. B., González-Hernández, F., Bustillos, R. O., & Reyes-Garcia, C. A. (2017) · Telematics and Informatics · doi:10.1016/j.tele.2017.03.005 · Cited by 72 · see citing works
Spontaneous facial expression database for academic emotion inference in online learning
Bian, C., Zhang, Y., Yang, F., Bi, W., & Lu, W. (2018) · IET Computer Vision · doi:10.1049/iet-cvi.2018.5281 · Cited by 62 · see citing works
A virtual environment for learning computer coding using gamification and emotion recognition
Cabada, R. Z., Estrada, M. L. B., Ríos-Félix, J. M., & Alor‐Hernández, G. (2018) · Interactive Learning Environments · doi:10.1080/10494820.2018.1558256 · Cited by 61 · see citing works
Richards, D., & Dignum, V. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12863 · Cited by 59 · see citing works
Detecting Student Engagement: Human Versus Machine
Bosch, N. (2016) · User Modeling, Adaptation, and Personalization · doi:10.1145/2930238.2930371 · Cited by 55 · see citing works
Self-citationMangaroska, K., Martínez-Maldonado, R., Vesin, B., & Gašević, D. (2021) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12542 · Cited by 52 · see citing works
Student Emotion, Co-Occurrence, and Dropout in a MOOC Context.
Dillon, J., Bosch, N., Chetlur, M., Wanigasekara, N., Ambrose, G., Sengupta, B., & D’Mello, S. K. (2016) · Educational Data Mining · Cited by 47 · see citing works
Self-citationJarrell, A., Harley, J. M., & Lajoie, S. P. (2016) · Journal of Computers in Education · doi:10.1007/s40692-016-0064-3 · Cited by 47 · see citing works
Toward active and unobtrusive engagement assessment of distance learners
Booth, B. M., Ali, A., Narayanan, S. S., Bennett, I., & Farag, A. A. (2017) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2017.8273641 · Cited by 45 · see citing works
Types of Game-Based Learning in Education: A brief state of the art and the implementation in Greece
Katerina, T., Dimitra, K., Kousaris, K., & Christina, Z. (2020) · The European Educational Researcher · doi:10.31757/euer.324 · Cited by 43 · see citing works
Kubsch, M., Krist, C., & Rosenberg, J. M. (2022) · Journal of Research in Science Teaching · doi:10.1002/tea.21803 · Cited by 42 · see citing works
Assessment Gaze, Refraction, and Blur
Baker, E., Chung, G. K. W. K., & Cai, L. (2016) · Review of Research in Education · doi:10.3102/0091732x16679806 · Cited by 8 · see citing works
Conati, C., Lallé, S., Rahman, M. A., & Toker, D. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3301400 · Cited by 39 · see citing works
Modeling Team-level Multimodal Dynamics during Multiparty Collaboration
Eloy, L., Stewart, A., Amon, M. J., Reinhardt, C., Michaels, A., Sun, C., Shute, V. J., Duran, N. D., & D’Mello, S. K. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3353748 · Cited by 39 · see citing works
Self-citationRamakrishnan, A., Zylich, B., Ottmar, E., LoCasale-Crouch, J., & Whitehill, J. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3059209 · Cited by 40 · see citing works
Salmeron-Majadas, S., Baker, R., Santos, O., & Boticario, J. (2018) · IEEE Access · doi:10.1109/access.2018.2854966 · Cited by 32 · see citing works
Self-citationReliability and validity of machine vision for the assessment of facial expressions
Beringer, M., Spohn, F., Hildebrandt, A., Wacker, J., & Recio, G. (2019) · Cognitive Systems Research · doi:10.1016/j.cogsys.2019.03.009 · Cited by 36 · see citing works
Diržytė, A., Vijaikis, A., Perminas, A., & Rimasiute-Knabikiene, R. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18179158 · Cited by 43 · see citing works
Accuracy vs. Availability Heuristic in Multimodal Affect Detection in the Wild
Bosch, N., Chen, H., D'Mello, S. K., Baker, R., & Shute, V. (2015) · International Conference on Multimodal Interaction · doi:10.1145/2818346.2820739 · Cited by 39 · see citing works
Self-citationRecognition of learning-centered emotions using a convolutional neural network
González-Hernández, F., Cabada, R. Z., Estrada, M. L. B., & Rodriguez-Rangel, H. (2018) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-169514 · Cited by 36 · see citing works
Hutt, S., Grafsgaard, J. F., & D'Mello, S. K. (2019) · International Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300726 · Cited by 48 · see citing works
Self-citationRezaee, M., Perumal, T., Shiri, F. M., & Ahmadi, E. (2024) · Journal on artificial intelligence · doi:10.32604/jai.2024.048911 · Cited by 33 · see citing works
Park, S., & Ryu, J. (2018) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2018.1469710 · Cited by 32 · see citing works
Dahlstrom‐Hakki, I., Asbell‐Clarke, J., & Rowe, E. A. (2018) · Mind Brain and Education · doi:10.1111/mbe.12177 · Cited by 32 · see citing works
“Engaged Faces”: Measuring and Monitoring Student Engagement from Face and Gaze Behavior
Carolis, B. N. D., D’Errico, F., Macchiarulo, N., & Palestra, G. (2019) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1145/3358695.3361748 · Cited by 32 · see citing works
Schrader, C., & Kalyuga, S. (2020) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2019.102374 · Cited by 31 · see citing works
Semi-supervised model personalization for improved detection of learner's emotional engagement
Alyüz, N., Okur, E., Oktay, E., Genc, U., Aslan, S., Mete, S. E., Bert Arnrich, A., & Esme, A. A. (2016) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2993148.2993166 · Cited by 31 · see citing works
Assessing the Effects of Open Models of Learning and Enjoyment in a Digital Learning Game
Hou, X., Nguyen, H. A., Richey, J. E., Harpstead, E., Hammer, J., & McLaren, B. M. (2021) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00250-6 · Cited by 33 · see citing works
Toward Automated Classroom Observation: Predicting Positive and Negative Climate
Ramakrishnan, A., Ottmar, E., LoCasale‐Crouch, J., & Whitehill, J. (2019) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2019.8756529 · Cited by 29 · see citing works
Detecting Learner Engagement in MOOCs using Automatic Facial Expression Recognition
Abhilash, D., & Gopalan, A. (2020) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon45650.2020.9125149 · Cited by 29 · see citing works
Towards developing a tool for UX evaluation using facial expression
Munim, K. M., Islam, I., Khatun, M., Karim, M. M., & Islam, M. N. (2017) · International Conference on Electrical Information and Communication Technology (EICT) · doi:10.1109/eict.2017.8275227 · Cited by 28 · see citing works
Ashwin, T. S., Prakash, V., & Rajendran, R. (2023) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2023.100125 · Cited by 33 · see citing works
Behavioral Engagement Detection of Students in the Wild
Okur, E., Alyüz, N., Aslan, S., Genc, U., Tanriover, C., & Esme, A. A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-61425-0_21 · Cited by 27 · see citing works
An unobtrusive and multimodal approach for behavioral engagement detection of students
Alyüz, N., Okur, E., Genc, U., Aslan, S., Tanriover, C., & Esme, A. A. (2017) · doi:10.1145/3139513.3139521 · Cited by 27 · see citing works
Multimodal, Multiparty Modeling of Collaborative Problem Solving Performance
Subburaj, S. K., Stewart, A., Rao, A., & D’Mello, S. K. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3418877 · Cited by 25 · see citing works
Self-citationPabba, C., & Kumar, P. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17533-w · Cited by 24 · see citing works
Possaghi, I., Vesin, B., Zhang, F.-R., Sharma, K., Knudsen, C., Bjørkum, H., & Papavlasopoulou, S. (2025) · Smart Learning Environments · doi:10.1186/s40561-025-00410-4 · Cited by 27 · see citing works
Automatic Facial Expression Analysis of Students in Teaching Environments
Tang, C., Xu, P., Luo, Z., Zhao, G., & Zou, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-25417-3_52 · Cited by 22 · see citing works
Recognizing Students' Emotions based on Facial Expression Analysis
Li, X., Yue, R., Jia, W., Wang, H., & Zheng, Y. (2021) · International Conference on Information Technology in Medicine and Education · doi:10.1109/itme53901.2021.00030 · Cited by 21 · see citing works
Towards More Robust Automatic Facial Expression Recognition in Smart Environments
Bernin, A., Müller, L., Ghose, S., von Luck, K., Grecos, C., Wang, Q., & Vogt, F. (2017) · Petra · doi:10.1145/3056540.3056546 · Cited by 21 · see citing works
Multimodal-multisensor affect detection
D’Mello, S. K., Bosch, N., & Chen, H. (2018) · ACM eBooks · doi:10.1145/3107990.3107998 · Cited by 19 · see citing works
Self-citationHarnessing Label Uncertainty to Improve Modeling: An Application to Student Engagement Recognition
Aung, A. M., & Whitehill, J. (2018) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2018.00033 · Cited by 19 · see citing works
Olugbade, T., Newbold, J. W., Johnson, R., Volta, E., Alborno, P., Niewiadomski, R., Dillon, M., Volpe, G., & Bianchi‐Berthouze, N. (2020) · Northumbria Research Link (Northumbria University) · Cited by 17 · see citing works
Predicting Student Engagement Using Sequential Ensemble Model
Tian, X., Nunes, B. P., Liu, Y., & Manrique, R. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3342860 · Cited by 17 · see citing works
A Comparison of Face-based and Interaction-based Affect Detectors in Physics Playground
Kai, S., Paquette, L., Baker, R. S., Bosch, N., D’Mello, S. K., Ocumpaugh, J. L., Shute, V. J., & Ventura, M. (2015) · Cited by 17 · see citing works
Self-citationMcCarthy, K. S., Crossley, S. A., Meyers, K., Boser, U., Allen, L. K., Chaudhri, V. K., Collins‐Thompson, K., D’Mello, S. K., Choudhury, M. D., Garg, K., et al. (2022) · Technology Mind and Behavior · doi:10.1037/tmb0000063 · Cited by 17 · see citing works
Self-citationBeyond Team Makeup: Diversity in Teams Predicts Valued Outcomes in Computer-Mediated Collaborations
Stewart, A., Amon, M. J., Duran, N. D., & D’Mello, S. K. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376279 · Cited by 17 · see citing works
Self-citationExtending Cognitive Load Theory: The CLAM Framework for Biometric, Adaptive, and Ethical Learning
Vasilaki, E., & Mavrogianni, A. (2025) · Psychology International · doi:10.3390/psycholint7020040 · Cited by 23 · see citing works
Effect of emotion-aware interventions on students’ behavioral and emotional states
Aslan, S., Alyüz, N., Okur, E., Mete, S. E., Oktay, E., & Esme, A. A. (2018) · Educational Technology Research and Development · doi:10.1007/s11423-018-9589-7 · Cited by 16 · see citing works
Multimodal Data Fusion to Track Students’ Distress during Educational Gameplay
Moon, J., Ke, F., Sokolikj, Z., & Dahlstrom‐Hakki, I. (2022) · Journal of Learning Analytics · doi:10.18608/jla.2022.7631 · Cited by 16 · see citing works
Karumbaiah, S., Baker, R., Tao, Y., & Liu, Z.-Y. (2022) · International Conference on Learning Analytics and Knowledge · doi:10.1145/3506860.3506863 · Cited by 21 · see citing works
Self-citationReasons and Responses: A Multimodal Serious Games Evaluation Framework
Shoukry, L., & Göbel, S. P. (2017) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2017.2737953 · Cited by 15 · see citing works
Gupta, S., Kumar, P., & Tekchandani, R. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-18021-x · Cited by 14 · see citing works
D’Mello, S. K., Duran, N., Michaels, A., & Stewart, A. E. B. (2024) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-023-09387-6 · Cited by 14 · see citing works
Self-citationOlugbade, T., Newbold, J. W., Johnson, R., Volta, E., Alborno, P., Niewiadomski, R., Dillon, M., Volpe, G., & Bianchi‐Berthouze, N. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.2978069 · Cited by 14 · see citing works
English language learner experiences of formal and informal learning environments
Epp, C. D. (2016) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2883851.2883896 · Cited by 14 · see citing works
Alyüz, N., Okur, E., Oktay, E., Genc, U., Aslan, S., Mete, S. E., Stanhill, D., Bert Arnrich, A., & Esme, A. A. (2016) · Cited by 14 · see citing works
Cabada, R. Z., Estrada, M. L. B., González-Hernández, F., & Rodriguez-Rangel, H. (2017) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2017.141 · Cited by 14 · see citing works
Using Body Language of Avatars in VR Meetings as Communication Status Cue
Kurzweg, M., Reinhardt, J., Nabok, W., & Wolf, K. (2021) · Mensch und Computer (MuC) · doi:10.1145/3473856.3473865 · Cited by 13 · see citing works
Affective effects of gamification
Korn, O., & Rees, A. (2019) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3316782.3316783 · Cited by 13 · see citing works
Rutherford, T. A., Rodrigues, A., Duque-Baird, S., Veng, S., Mykyta‐Chomsky, R., Cao, Y., Chisholm, K., & Bergwall, E. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100367 · Cited by 15 · see citing works
In-classroom self-tracking for teachers and students
Lascio, E. D., Gashi, S., Krasic, D., & Silvia Santini, B. (2017) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3123024.3125504 · Cited by 12 · see citing works
Multimodal Affect Detection in the Wild
Bosch, N. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2823316 · Cited by 12 · see citing works
Self-citationFacial Expressions of Comprehension (FEC)
Turan, C., Neergaard, K. D., & Lam, K. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2954498 · Cited by 13 · see citing works
Optimizing student engagement detection using facial and behavioral features
Das, R., & Dev, S. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11317-z · Cited by 11 · see citing works
Measuring student behavioral engagement using histogram of actions
Abdelkawy, A., Farag, A. A., Alkabbany, I. M., Ali, A., Foreman, C., Tretter, T. R., & Hindy, N. C. (2024) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.11.002 · Cited by 11 · see citing works
Identification of Low-engaged Learners in Robot-led Second Language Conversations with Adults
Engwall, O., Cumbal, R., Lopes, J., Ljung, M., & Månsson, L. (2022) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3503799 · Cited by 11 · see citing works
Can a computer administer a Wechsler Intelligence Test
Vrana, S., & Vrana, D. T. (2017) · Professional Psychology Research and Practice · doi:10.1037/pro0000128 · Cited by 11 · see citing works
Building a Corpus and a Local Binary Pattern Recognizer for Learning-Centered Emotions
Cabada, R. Z., Estrada, M. L. B., González-Hernández, F., Bustillos, R. O., Alor‐Hernández, G., & Reyes-Garcia, C. A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-62428-0_43 · Cited by 11 · see citing works
Marlow, J., Wiese, J., & Avrahami, D. (2017) · International Conference on Communities and Technologies (C&T) · doi:10.1145/3083671.3083672 · Cited by 10 · see citing works
Temporal Generalizability of Face-Based Affect Detection in Noisy Classroom Environments
Bosch, N., D’Mello, S. K., Baker, R. S., Ocumpaugh, J. L., & Shute, V. J. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-19773-9_5 · Cited by 10 · see citing works
Self-citationDetecting stress from mouse-gaze attraction
Wang, J., Fu, E. Y., Ngai, G., Leong, H. V., & Huang, M. X. (2019) · ACM Symposium on Applied Computing · doi:10.1145/3297280.3297345 · Cited by 8 · see citing works
Distinguishing Anxiety Subtypes of English Language Learners Towards Augmented Emotional Clarity
Lee, H., Mandalapu, V., Kleinsmith, A., & Gong, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-52240-7_29 · Cited by 8 · see citing works
The AffectMove 2021 Challenge - Affect Recognition from Naturalistic Movement Data
Olugbade, T., Sagoleo, R., Ghisio, S., Gold, N., Williams, A. C. D. C., de Gelder, B., Camurri, A., Volpe, G., & Bianchi‐Berthouze, N. (2021) · International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw52867.2021.9666322 · Cited by 7 · see citing works
Multi-modal affect detection for learning applications
Gogia, Y., Singh, E., Mohatta, S., & Sreejith (2016) · IEEE Region 10 Conference · doi:10.1109/tencon.2016.7848760 · Cited by 7 · see citing works
Automatic engagement detection in the education: critical review
Kasatkina, D., Kravchenko, A., Kupriyanov, R., & Nekhorosheva, E. (2020) · Современная зарубежная психология · doi:10.17759/jmfp.2020090305 · Cited by 9 · see citing works
Multi-label Disengagement and Behavior Prediction in Online Learning
Verma, M., Nakashima, Y., Takemura, N., & Nagahara, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-11644-5_60 · Cited by 7 · see citing works
Leveraging part-and-sensitive attention network and transformer for learner engagement detection
Su, R., He, L., & Luo, M. (2024) · Alexandria Engineering Journal · doi:10.1016/j.aej.2024.06.074 · Cited by 6 · see citing works
Perspectives de l'OCDE sur l'éducation numérique 2021
OCDE (2022) · Digital education outlook · doi:10.1787/d5fe6bd0-fr · Cited by 19 · see citing works
Protocol for Eliciting Driver Frustration in an In-vehicle Environment
Joshi, A., Attia, Y., & Mishra, T. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925489 · Cited by 6 · see citing works
Validation of Emotions as a Measure of Selective Attention in Children with Autism Spectrum Disorder
Banire, B., Al‐Thani, D., & Qaraqe, M. (2020) · International Conference on Educational and Information Technology · doi:10.1145/3383923.3383954 · Cited by 6 · see citing works
González-Meneses, Y. N., Guerrero-García, J., Reyes-Garcia, C. A., Olmos-Pineda, I., & González-Calleros, J. M. (2019) · Communications in computer and information science · doi:10.1007/978-3-030-37386-3_16 · Cited by 6 · see citing works
Students' emotional self-labels for personalized models
Aslan, S., Okur, E., Alyüz, N., Mete, S. E., Oktay, E., Genc, U., & Esme, A. A. (2017) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3027385.3029452 · Cited by 6 · see citing works
- Authors
- Bosch, N., D'Mello, S., Baker, R., Ocumpaugh, J., Shute, V., Ventura, M., Wang, L., & Zhao, W.
- Affiliations
- University of Notre Dame (US)
- University of Notre Dame, Notre Dame, IN, USA
- Columbia University (US)
- Florida State University (US)
- Venue
- Proceedings of the 20th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2015-03-18
- Pages
- 379-388
- OpenAlex
- W2115417283
- Cited by
- 178
- References
- 45
- Citations by year
- 2026: 12 · 2025: 19 · 2024: 11 · 2023: 11 · 2022: 11 · 2021: 14 · 2020: 17 · 2019: 20 · 2018: 18 · 2017: 21 · 2016: 15 · 2015: 9
- FWCI
- 28.05
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Funders
- NSF (DRL 1235958)
- Bill & Melinda Gates Foundation
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- affective computing, affect recognition, game-based learning, facial expression recognition, body movement analysis, off-task behavior, boredom, confusion, engagement, frustration, delight, classroom climate
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
