The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Classifying Emotions and Engagement in Online Learning Based on a Single Facial Expression Recognition Neural Network
Abstract
In this article, behaviour of students in the e-learning environment is analyzed. The novel pipeline is proposed based on video facial processing. At first, face detection, tracking and clustering techniques are applied to extract the sequences of faces of each student. Next, a single efficient neural network is used to extract emotional features in each frame. This network is pre-trained on face identification and fine-tuned for facial expression recognition on static images from AffectNet using a specially developed robust optimization technique. It is shown that the resulting facial features can be used for fast simultaneous prediction of students’ engagement levels (from disengaged to highly engaged), individual emotions (happy, sad, etc.,) and group-level affect (positive, neutral or negative). This model can be used for real-time video processing even on a mobile device of each student without the need for sending their facial video to the remote server or teacher's PC. In addition, the possibility to prepare a summary of a lesson is demonstrated by saving short clips of different emotions and engagement of all students. The experimental study on the datasets from EmotiW (Emotion Recognition in the Wild) challenges showed that the proposed network significantly outperforms existing single models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
VGGFace2: A Dataset for Recognising Faces across Pose and Age
Cao, Q., Shen, L., Xie, W., Parkhi, O. M., & Zisserman, A. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00020 · Cited by 3,032
AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2740923 · Cited by 2,338
Openface 2.0: facial behavior analysis toolkit
Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L.-P. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00019 · Cited by 1,861
The Faces of Engagement: Automatic Recognition of Student Engagementfrom Facial Expressions
Whitehill, J., Serpell, Z., Lin, Y.-C., Foster, A., & Movellan, J. R. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2316163 · Cited by 677
Affective Quality Attributed to Environments: A Factor Analytic Study
Russell, J. A., Ward, L. M., & Pratt, G. (1981) · Environment and Behavior · doi:10.1177/0013916581133001 · Cited by 371
Engagement detection in online learning: a review
Dewan, M. A. A., Murshed, M., & Lin, F. (2019) · Smart Learning Environments · doi:10.1186/s40561-018-0080-z · Cited by 350
Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion
Ma, F., Sun, B., & Li, S. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3122146 · Cited by 327
Facial Expression Recognition in the Wild via Deep Attentive Center Loss
Farzaneh, A. H., & Qi, X. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00245 · Cited by 326
TransFER: Learning Relation-aware Facial Expression Representations with Transformers
Xue, F., Wang, Q., & Guo, G. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00358 · Cited by 321
Distract Your Attention: Multi-Head Cross Attention Network for Facial Expression Recognition
Wen, Z., Lin, W., Wang, T., & Xu, G. (2023) · Biomimetics · doi:10.3390/biomimetics8020199 · Cited by 313
Leveraging recent advances in deep learning for audio-Visual emotion recognition
Schoneveld, L., Othmani, A., & Abdelkawy, H. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.03.007 · Cited by 237
Prediction and Localization of Student Engagement in the Wild
Kaur, A., Mustafa, A., Mehta, L., & Dhall, A. (2018) · International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta.2018.8615851 · Cited by 210
Frame Attention Networks for Facial Expression Recognition in Videos
Meng, D., Peng, X., Wang, K., & Qiao, Y. (2019) · 2019 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2019.8803603 · Cited by 196
Application of Deep Learning on Student Engagement in e-learning environments
Bhardwaj, P., Gupta, P. K., Panwar, H., Siddiqui, M. K., Morales-Menendez, R., & Bhaik, A. (2021) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2021.107277 · Cited by 182
A survey of emotion recognition methods with emphasis on E-Learning environments
Imani, M., & Montazer, G. A. (2019) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.102423 · Cited by 181
Savchenko, A. V. (2021) · 2021 IEEE 19th International Symposium on Intelligent Systems and Informatics (SISY) · doi:10.1109/sisy52375.2021.9582508 · Cited by 172
Self-citationEmotion recognition in the wild from videos using images
Bargal, S. A., Barsoum, E., Ferrer, C. C., & Zhang, C. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2997627 · Cited by 171
Zhang, Z., Li, Z., Liu, H., Cao, T., & Liu, S. (2019) · Journal of Educational Computing Research · doi:10.1177/0735633119825575 · Cited by 131
Sathik, M. M., & Jonathan, S. G. (2013) · SpringerPlus · doi:10.1186/2193-1801-2-455 · Cited by 94
Savchenko, A. (2019) · PeerJ Computer Science · doi:10.7717/peerj-cs.197 · Cited by 91
Self-citationEmotionCues : Emotion-Oriented Visual Summarization of Classroom Videos
Zeng, H., Shu, X., Wang, Y., Wang, Y., Zhang, L., Pong, T.-C., & Qu, H. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2963659 · Cited by 91
Assessing learning engagement based on facial expression recognition in MOOC’s scenario
Shen, J., Yang, H., Li, J., & Cheng, Z. (2021) · Multimedia Systems · doi:10.1007/s00530-021-00854-x · Cited by 89
EmotiW 2019: Automatic Emotion, Engagement and Cohesion Prediction Tasks
Dhall, A. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3355710 · Cited by 84
Viewing Student Affect and Learning through Classroom Observation and Physical Sensors
Dragon, T., Arroyo, I., Woolf, B. P., Burleson, W., el Kaliouby, R., & Eydgahi, H. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-69132-7_8 · Cited by 80
Yang, J., Wang, K., Peng, X., & Qiao, Y. (2018) · Proceedings of the 20th ACM International Conference on Multimodal Interaction · doi:10.1145/3242969.3264981 · Cited by 79 · cited 2×
“The best results on the testing set in the 2018 challenge were obtained by additional LBP-TOP facial descriptor and C3D action features [43].”
MethodInfluentialAshwin, T. S., & Guddeti, R. M. R. (2020) · Future Generation Computer Systems · doi:10.1016/j.future.2020.02.075 · Cited by 79
Automatic Engagement Prediction with GAP Feature
Niu, X., Han, H., Zeng, J., Sun, X., Shan, S., Huang, Y., Yang, S., & Chen, X. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3264982 · Cited by 77
Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognition
Zhou, H., Meng, D., Zhang, Y., Peng, X., Du, J., Wang, K., & Qiao, Y. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3355713 · Cited by 75
Antoniadis, P., Filntisis, P. P., & Maragos, P. A. (2021) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg52635.2021.9667014 · Cited by 70
Liu, T., Wang, J., Yang, B., & Wang, X. (2020) · Infrared Physics & Technology · doi:10.1016/j.infrared.2020.103594 · Cited by 67
Student engagement with mobile‐based assessment systems: A survival analysis
Bacca-Acosta, J., & Avila, C. (2020) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12475 · Cited by 62
Noisy Student Training Using Body Language Dataset Improves Facial Expression Recognition
Kumar, V., Rao, S., & Yu, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-66415-2_53 · Cited by 56
Wang, X., Liu, T., Wang, J., & Tian, J. (2021) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2021.1938389 · Cited by 55
Video-based Frame-level Facial Analysis of Affective Behavior on Mobile Devices using EfficientNets
Savchenko, A. V. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00263 · Cited by 50
Self-citationAudio-Visual Automatic Group Affect Analysis
Sharma, G., Dhall, A., & Cai, J. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3104170 · Cited by 49
Predicting Engagement Intensity in the Wild Using Temporal Convolutional Network
Thomas, C., Nair, N., & Jayagopi, D. B. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3264984 · Cited by 47
Bi-modality Fusion for Emotion Recognition in the Wild
Li, S., Zheng, W., Zong, Y., Lu, C., Tang, C., Jiang, X., Liu, J., & Xia, W. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3355719 · Cited by 47
Multi-rate Attention Based GRU Model for Engagement Prediction
Zhu, B., Lan, X., Guo, X., Barner, K. E., & Boncelet, C. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3417965 · Cited by 47
Low-resolution facial expression recognition: A filter learning perspective
Yan, Y., Zhang, Z., Chen, S., & Wang, H. (2019) · Signal Processing · doi:10.1016/j.sigpro.2019.107370 · Cited by 45
Group Level Audio-Video Emotion Recognition Using Hybrid Networks
Liu, C., Jiang, W., Wang, M., & Tang, T. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3417968 · Cited by 37
Students behavioural analysis in an online learning environment using data mining
Ratnapala, I. P., Ragel, R., & Deegalla, S. (2014) · International Conference on Information and Automation for Sustainability (ICIAfS) · doi:10.1109/iciafs.2014.7069609 · Cited by 35
FaceEngage: Robust Estimation of Gameplay Engagement from User-Contributed (YouTube) Videos
Chen, X., Niu, L., Veeraraghavan, A., & Sabharwal, A. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2945014 · Cited by 33
Demochkina, P., & Savchenko, A. V. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68821-9_25 · Cited by 32
Self-citationImplicit Knowledge Injectable Cross Attention Audiovisual Model for Group Emotion Recognition
Wang, Y., Wu, J., Heracleous, P., Wada, S., Kimura, R., & Kurihara, S. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3417960 · Cited by 29
Preference prediction based on a photo gallery analysis with scene recognition and object detection
Savchenko, A., Demochkin, K. V., & Grechikhin, I. S. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108248 · Cited by 29
Self-citationBootstrap Model Ensemble and Rank Loss for Engagement Intensity Regression
Wang, K., Yang, J., Guo, D., Zhang, K., Peng, X., & Qiao, Y. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3355711 · Cited by 24
Audiovisual Classification of Group Emotion Valence Using Activity Recognition Networks
Pinto, J. R., Gonçalves, T., Pinto, C., Sanhudo, L., Fonseca, J., Gonçalves, F., Carvalho, P., & Cardoso, J. S. (2020) · IEEE International Conference on Image Processing, Applications and Systems (IPAS) · doi:10.1109/ipas50080.2020.9334943 · Cited by 21
Multi-modal Fusion Using Spatio-temporal and Static Features for Group Emotion Recognition
Sun, M., Li, J., Feng, H., et al. (2020) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3382507.3417971 · Cited by 16
Estimating Student Learning Affect Using Facial Emotions
Zakka, B. E., & Vadapalli, H. (2020) · International Multidisciplinary Information Technology and Engineering Conference (IMITEC) · doi:10.1109/imitec50163.2020.9334075 · Cited by 16
Wu, J., Zhou, Z., Wang, Y., Li, Y., Xu, X., & Uchida, Y. (2019) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3340555.3355717 · Cited by 13
Tomczyk, Ł., Potyrała, K., Demeshkant, N., & Czerwiec, K. (2021) · Iberian Conference on Information Systems and Technologies (CISTI) · doi:10.23919/cisti52073.2021.9476521 · Cited by 10
Metadata
- Authors
- Savchenko, A. V., Savchenko, L. V., & Makarov, I.
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2022-10-01
- Volume
- 13
- Issue
- 4
- Pages
- 2132-2143
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W4285250231
- Cited by
- 346
- References
- 53
- FWCI
- 61.05
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.