The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Automated Detection of Engagement Using Video-Based Estimation of Facial Expressions and Heart Rate
Abstract
We explored how computer vision techniques can be used to detect engagement while students (N = 22) completed a structured writing activity (draft-feedback-review) similar to activities encountered in educational settings. Students provided engagement annotations both concurrently during the writing activity and retrospectively from videos of their faces after the activity. We used computer vision techniques to extract three sets of features from videos, heart rate, Animation Units (from Microsoft Kinect Face Tracker), and local binary patterns in three orthogonal planes (LBP-TOP). These features were used in supervised learning for detection of concurrent and retrospective self-reported engagement. Area under the ROC Curve (AUC) was used to evaluate classifier accuracy using leave-several-students-out cross validation. We achieved an AUC = .758 for concurrent annotations and AUC = .733 for retrospective annotations. The Kinect Face Tracker features produced the best results among the individual channels, but the overall best results were found using a fusion of channels.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
PSYCHOLOGICAL CONDITIONS OF PERSONAL ENGAGEMENT AND DISENGAGEMENT AT WORK.
Kahn, W. A. (1990) · Academy of Management Journal · doi:10.2307/256287 · Cited by 10,975
Independent component analysis, A new concept?
Comon, P. (1994) · Signal Processing · doi:10.1016/0165-1684(94)90029-9 · Cited by 8,205
A comparative study of texture measures with classification based on featured distributions
Ojala, T., Pietikäinen, M., & Harwood, D. (1996) · Pattern Recognition · doi:10.1016/0031-3203(95)00067-4 · Cited by 6,882
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019
Autonomic nervous system activity in emotion: a review
Kreibig, S. D. (2010) · Biological Psychology · doi:10.1016/j.biopsycho.2010.03.010 · Cited by 3,188 · cited 1×
“The present emphasis is on physiology-based and facial-feature based engagement detection as these are the two methodswe explore in this research.”
BackgroundHandbook of Research on Student Engagement
(2012) · doi:10.1007/978-1-4614-2018-7 · Cited by 3,050
Estimating attributes: Analysis and extensions of RELIEF
Kononenko, I. (1994) · Lecture Notes in Computer Science · doi:10.1007/3-540-57868-4_57 · Cited by 2,958
Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
Zhao, G., & Pietikainen, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1110 · Cited by 2,846
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Facial expression recognition based on Local Binary Patterns: A comprehensive study
Shan, C., Gong, S., & McOwan, P. W. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.08.005 · Cited by 2,300
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 2×
“Classification accuracies between 84 and 99 percent have been achieved for detecting driver drowsiness detection using EEG methods [28].”
BackgroundMethodSelf-citationRemote plethysmographic imaging using ambient light
Verkruysse, W., Svaasand, L. O., & Nelson, J. S. (2008) · Optics Express · doi:10.1364/oe.16.021434 · Cited by 1,719
Non-contact, automated cardiac pulse measurements using video imaging and blind source separation
Poh, M.-Z., McDuff, D. J., & Picard, R. W. (2010) · Optics Express · doi:10.1364/oe.18.010762 · Cited by 1,643
Agency as a fourth aspect of students’ engagement during learning activities
Reeve, J., & Tseng, C.-M. (2011) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2011.05.002 · Cited by 1,618
Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam
Poh, M.-Z., McDuff, D. J., & Picard, R. W. (2011) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2010.2086456 · Cited by 1,486
Voluntary Facial Action Generates Emotion‐Specific Autonomic Nervous System Activity
Levenson, R. W., Ekman, P., & Friesen, W. V. (1990) · Psychophysiology · doi:10.1111/j.1469-8986.1990.tb02330.x · Cited by 1,248
High-Order Contrasts for Independent Component Analysis
Cardoso, J.-F. (1999) · Neural Computation · doi:10.1162/089976699300016863 · Cited by 1,213
An advanced detrending method with application to HRV analysis
Tarvainen, M. P., Ranta-aho, P. O., & Karjalainen, P. A. (2002) · IEEE Transactions on Biomedical Engineering · doi:10.1109/10.979357 · Cited by 1,131
THE ROLE OF SELF-EFFICACY BELIEFS INSTUDENT ENGAGEMENT AND LEARNING INTHECLASSROOM
Linnenbrink, E. A., & Pintrich, P. R. (2003) · Reading & Writing Quarterly · doi:10.1080/10573560308223 · Cited by 1,106
WEKA: a machine learning workbench
Holmes, G., Donkin, A., & Witten, I. H. (2002) · Proceedings of ANZIIS '94 - Australian New Zealnd Intelligent Information Systems Conference · doi:10.1109/anziis.1994.396988 · Cited by 930
Facing Imbalanced Data--Recommendations for the Use of Performance Metrics
Jeni, L. A., Cohn, J. F., & De La Torre, F. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.47 · Cited by 790
Detecting Driver Drowsiness Based on Sensors: A Review
Sahayadhas, A., Sundaraj, K., & Murugappan, M. (2012) · Sensors · doi:10.3390/s121216937 · Cited by 782
The Faces of Engagement: Automatic Recognition of Student Engagementfrom Facial Expressions
Whitehill, J., Serpell, Z., Lin, Y.-C., Foster, A., & Movellan, J. R. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2316163 · Cited by 677
Pantic, M., & Patras, I. (2006) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.859075 · Cited by 580
The Oxford Handbook of Affective Computing
(2015) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199942237.001.0001 · Cited by 482
Applying neural network analysis on heart rate variability data to assess driver fatigue
Patel, M., Lal, S. K. L., Kavanagh, D., & Rossiter, P. (2011) · Expert Systems with Applications · doi:10.1016/j.eswa.2010.12.028 · Cited by 418
Facial Action Recognition for Facial Expression Analysis From Static Face Images
Pantic, M., & Rothkrantz, L. J. M. (2004) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2004.825931 · Cited by 417
Contact-Free Measurement of Cardiac Pulse Based on the Analysis of Thermal Imagery
Garbey, M., Sun, N., Merla, A., & Pavlidis, I. (2007) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2007.891930 · Cited by 402
Positive Computing: Technology for Wellbeing and Human Potential
Calvo, R. A., & Peters, D. (2014) · doi:10.7551/mitpress/9764.001.0001 · Cited by 376 · cited 1×
“Index Terms—Engagement detection, remote heart rate measurement, facial expression, writing task Ç”
BackgroundSelf-citationCan SVM be used for automatic EEG detection of drowsiness during car driving?
Yeo, M. V. M., Li, X., Shen, K., & Wilder-Smith, E. P. V. (2009) · Safety Science · doi:10.1016/j.ssci.2008.01.007 · Cited by 368
Dynamics of facial expression extracted automatically from video
Littlewort, G., Bartlett, M. S., Fasel, I., Susskind, J., & Movellan, J. (2006) · Image and Vision Computing · doi:10.1016/j.imavis.2005.09.011 · Cited by 358
A real-time automated system for the recognition of human facial expressions
Anderson, K., & McOwan, P. W. (2006) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.854502 · Cited by 355
Heart rate measurement based on a time-lapse image.
Takano, C., & Ohta, Y. (2007) · Medical Engineering & Physics · doi:10.1016/j.medengphy.2006.09.006 · Cited by 345
Multimodal affect recognition in learning environments
Kapoor, A., & Picard, R. W. (2005) · Proceedings of the 13th annual ACM international conference on Multimedia · doi:10.1145/1101149.1101300 · Cited by 339
Meta-Analysis of the First Facial Expression Recognition Challenge
Valstar, M. F., Mehu, M., Jiang, B., Pantic, M., & Scherer, K. (2012) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2012.2200675 · Cited by 297
Fully Automatic Facial Action Recognition in Spontaneous Behavior
Bartlett, M. S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., & Movellan, J. (2006) · 7th International Conference on Automatic Face and Gesture Recognition (FGR06) · doi:10.1109/fgr.2006.55 · Cited by 290
An X-Band Microwave Life-Detection System
Chen, K., Misra, D. K., Wang, H., Chuang, H.-R., & Postow, E. (1986) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.1986.325760 · Cited by 284
Positive computing: technology for wellbeing and human potential
(2015) · Choice Reviews Online · doi:10.5860/choice.189530 · Cited by 283
Evaluation of an adaptive automation system using three EEG indices with a visual tracking task
Freeman, F. G., Mikulka, P. J., Prinzel, L. J., & Scerbo, M. W. (1999) · Biological Psychology · doi:10.1016/s0301-0511(99)00002-2 · Cited by 282
Radar remote monitoring of vital signs
Li, C., Cummings, J., Lam, J., Graves, E., & Wu, W. (2009) · IEEE Microwave Magazine · doi:10.1109/mmm.2008.930675 · Cited by 255
Recognizing engagement in human-robot interaction
Rich, C., Ponsler, B., Holroyd, A., & Sidner, C. L. (2010) · 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2010.5453163 · Cited by 252
Tian, Y.-L., Kanade, T., & Cohn, J. F. (2003) · Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition · doi:10.1109/afgr.2002.1004159 · Cited by 223
Thermistor at a Distance: Unobtrusive Measurement of Breathing
Fei, J., & Pavlidis, I. (2010) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2009.2032415 · Cited by 201
Learning From Examples in the Small Sample Case: Face Expression Recognition
Guo, G., & Dyer, C. R. (2005) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.846658 · Cited by 194
ENCARA2: Real-time detection of multiple faces at different resolutions in video streams
Castrillón, M., Déniz, O., Guerra, C., & Hernández, M. (2007) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2006.11.004 · Cited by 184
A Machine Learning Approach to Improve Contactless Heart Rate Monitoring Using a Webcam
Monkaresi, H., Calvo, R. A., & Yan, H. (2014) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2013.2291900 · Cited by 179
Self-citationNew Perspectives on Affect and Learning Technologies
(2011) · Springer eBooks · doi:10.1007/978-1-4419-9625-1 · Cited by 173
Automatically Recognizing Facial Expression: Predicting Engagement and Frustration
Grafsgaard, J. F., Wiggins, J. B., Boyer, K. E., Wiebe, E., & Lester, J. C. (2013) · Educational Data Mining · Cited by 171
Estimating user's engagement from eye-gaze behaviors in human-agent conversations
Nakano, Y., & Ishii, R. (2010) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1719970.1719990 · Cited by 166
Investigating Spontaneous Facial Action Recognition through AAM Representations of the Face
Lucey, S., Bilal, A., & F., J. (2007) · Face Recognition · doi:10.5772/4841 · Cited by 161
Valstar, M., & Pantić, M. (2007) · Springer eBooks · doi:10.1007/978-3-540-75773-3_13 · Cited by 160
Radar sensing of heartbeat and respiration at a distance with applications of the technology
Greneker, E. F. (1997) · doi:10.1049/cp:19971650 · Cited by 149
Models for multiparty engagement in open-world dialog
Bohus, D., & Horvitz, E. (2009) · Proceedings of the SIGDIAL 2009 Conference on The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue - SIGDIAL '09 · doi:10.3115/1708376.1708409 · Cited by 145
Facial Features for Affective State Detection in Learning Environments
McDaniel, B., D’Mello, S. K., King, B. G., Chipman, P., Tapp, K. M., & Graesser, A. C. (2007) · Proceedings of the Annual Meeting of the Cognitive Science Society · Cited by 132
Self-citationDetection of Emotions during Learning with AutoTutor
Chipman, P., D’Mello, S. K., Gholson, B., Graesser, A. C., McDaniel, B., & Witherspoon, A. M. (2006) · Proceedings of the Annual Meeting of the Cognitive Science Society · Cited by 120
Self-citationAn exploration of user engagement in HCI
Peters, C., Castellano, G., & de Freitas, S. (2009) · doi:10.1145/1655260.1655269 · Cited by 114
D’Mello, S. K., & Kory-Westlund, J. M. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388686 · Cited by 103
Self-citationBeyond the basic emotions: what should affective computing compute?
D’Mello, S. K., & Calvo, R. A. (2013) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2468356.2468751 · Cited by 103
Self-citationAffect Detection from Multichannel Physiology during Learning Sessions with AutoTutor
Hussain, M. S., AlZoubi, O., Calvo, R. A., & D’Mello, S. K. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21869-9_19 · Cited by 97
Self-citationCrowdsourcing for Affective Annotation of Video: Development of a Viewer-reported Boredom Corpus
Soleymani, M., & Larson, M. (2010) · Archive ouverte UNIGE (University of Geneva) · Cited by 83
A Generic Forecast Verification Framework for Administrative Purposes
Mason, S. J., & Weigel, A. P. (2008) · Monthly Weather Review · doi:10.1175/2008mwr2553.1 · Cited by 78
3D Corpus of Spontaneous Complex Mental States
Mahmoud, M., Baltrušaitis, T., Robinson, P., & Riek, L. D. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24600-5_24 · Cited by 73
Learning-based heart rate detection from remote photoplethysmography features
Hsu, Y., Lin, Y.-L., & Hsu, W. H. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854440 · Cited by 71
Liang, W. C., Yuan, J. S. C., Sun, D. C., & Lin, M. H. (2009) · Sensors · doi:10.3390/s90906913 · Cited by 70
An Automated Optimal Engagement and Attention Detection System Using Electrocardiogram
Belle, A., Hargraves, R. H., & Najarian, K. (2012) · Computational and Mathematical Methods in Medicine · doi:10.1155/2012/528781 · Cited by 61
Stevens, R. H., Galloway, T. L., & Berka, C. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73078-1_22 · Cited by 57
Natural affect data — Collection & annotation in a learning context
Afzal, S., & Robinson, P. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349537 · Cited by 48
Emotions while writing about emotional and non-emotional topics
D’Mello, S. K., & Mills, C. (2013) · Motivation and Emotion · doi:10.1007/s11031-013-9358-1 · Cited by 46
Self-citationEmbodied Affect in Tutorial Dialogue: Student Gesture and Posture
Grafsgaard, J. F., Wiggins, J. B., Boyer, K. E., Wiebe, E., & Lester, J. C. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39112-5_1 · Cited by 43
When Does Disengagement Correlate with Performance in Spoken Dialog Computer Tutoring?
Forbes-Riley, K., & Litman, D. J. (2013) · International Journal of Artificial Intelligence in Education · doi:10.3233/jai-130028 · Cited by 39
Towards an Optimal Affect-Sensitive Instructional System of cognitive skills
Whitehill, J., Serpell, Z. N., Foster, A., Lin, Y.-C., Pearson, B., Bartlett, M. S., & Movellan, J. R. (2011) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2011.5981778 · Cited by 31
When Does Disengagement Correlate with Learning in Spoken Dialog Computer Tutoring?
Forbes-Riley, K., & Litman, D. J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21869-9_13 · Cited by 17
Automatic Gaze-Based Detection of Mind Wandering during Reading.
D’Mello, S. K., Cobian, J., & Hunter, M. B. (2013) · Cited by 16
Self-citationAutomatic natural expression recognition using head movement and skin color features
Monkaresi, H., Calvo, R. A., & Hussain, M. S. (2012) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/2254556.2254678 · Cited by 12
Self-citationA Dynamic Approach for Detecting Naturalistic Affective States from Facial Videos during HCI
Monkaresi, H., Hussain, M., & Calvo, R. (2012) · Australasian Conference on Artificial Intelligence · doi:10.1007/978-3-642-35101-3_15 · Cited by 5
Self-citation
Cited by
Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J. F., & Sebe, N. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.263 · Cited by 340
Shoumy, N. J., Ang, L.-M., Seng, K. P., Rahaman, D. M. M., & Zia, T. (2020) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2019.102447 · Cited by 171
Mehta, N. K., Prasad, S. S., Saurav, S., Saini, R., & Singh, S. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03200-4 · Cited by 97
Kharbat, F. F., Alshawabkeh, A., & Woolsey, M. L. (2020) · Aslib Journal of Information Management · doi:10.1108/ajim-02-2020-0054 · Cited by 63
RAISE: Robotics & AI to improve STEM and social skills for elementary school students
Hughes, C. E., Dieker, L. A., Glavey, E. M., Hines, R. A., Wilkins, I., Ingraham, K., Bukaty, C. A., Ali, K., Shah, S., Murphy, J., et al. (2022) · Frontiers in Virtual Reality · doi:10.3389/frvir.2022.968312 · Cited by 47
Metadata
- Authors
- Monkaresi, H., Bosch, N., Calvo, R. A., & D'Mello, S. K.
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2017-01-01
- Volume
- 8
- Issue
- 1
- Pages
- 15-28
- ISSN
- 1949-3045 (ISSN-L)
- OpenAlex
- W2345031027
- Cited by
- 329
- References
- 76
- FWCI
- 33.14
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.