The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
The Faces of Engagement: Automatic Recognition of Student Engagementfrom Facial Expressions
Abstract
Student engagement is a key concept in contemporary education, where it is valued as a goal in its own right. In this paper we explore approaches for automatic recognition of engagement from students' facial expressions. We studied whether human observers can reliably judge engagement from the face; analyzed the signals observers use to make these judgments; and automated the process using machine learning. We found that human observers reliably agree when discriminating low versus high degrees of engagement (Cohen's κ = 0.96). When fine discrimination is required (four distinct levels) the reliability decreases, but is still quite high ( κ = 0.56). Furthermore, we found that engagement labels of 10-second video clips can be reliably predicted from the average labels of their constituent frames (Pearson r=0.85), suggesting that static expressions contain the bulk of the information used by observers. We used machine learning to develop automatic engagement detectors and found that for binary classification (e.g., high engagement versus low engagement), automated engagement detectors perform with comparable accuracy to humans. Finally, we show that both human and automatic engagement judgments correlate with task performance. In our experiment, student post-test performance was predicted with comparable accuracy from engagement labels ( r=0.47) as from pre-test scores ( r=0.44).
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
School Engagement: Potential of the Concept, State of the Evidence
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004) · Review of Educational Research · doi:10.3102/00346543074001059 · Cited by 12,581
Friedman, J., Hastie, T., & Tibshirani, R. (2000) · The Annals of Statistics · doi:10.1214/aos/1016218223 · Cited by 6,884
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303
Acquisition of cognitive skill
Anderson, J. R. (1982) · Psychological Review · doi:10.1037/0033-295x.89.4.369 · Cited by 3,663
Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions
Zhao, G., & Pietikainen, M. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1110 · Cited by 2,846
Comprehensive database for facial expression analysis
Kanade, T., Cohn, J. F., & Tian, Y. (2002) · Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) · doi:10.1109/afgr.2000.840611 · Cited by 2,591
A desicion-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1995) · Lecture Notes in Computer Science · doi:10.1007/3-540-59119-2_166 · Cited by 2,252
Distortion invariant object recognition in the dynamic link architecture
Lades, M., Vorbruggen, J. C., Buhmann, J., Lange, J., von der Malsburg, C., Wurtz, R. P., & Konen, W. (1993) · IEEE Transactions on Computers · doi:10.1109/12.210173 · Cited by 1,805
Student engagement in high school classrooms from the perspective of flow theory
Shernoff, D. J., Csikszentmihalyi, M., Shneider, B., & Shernoff, E. S. (2003) · School Psychology Quarterly · doi:10.1521/scpq.18.2.158.21860 · Cited by 1,237
Intelligent Tutoring Goes To School in the Big City
Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (2018) · doi:10.1184/r1/6470153 · Cited by 1,105
The development and evaluation of a survey to measure user engagement
O'Brien, H. L., & Toms, E. G. (2010) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21229 · Cited by 790
Biocybernetic system evaluates indices of operator engagement in automated task
Pope, A. T., Bogart, E. H., & Bartolome, D. S. (1995) · Biological Psychology · doi:10.1016/0301-0511(95)05116-3 · Cited by 663
Matthews, G., Campbell, S. E., Falconer, S., Joyner, L. A., Huggins, J., Gilliland, K., Grier, R. A., & Warm, J. S. (2002) · Emotion · doi:10.1037//1528-3542.2.4.315 · Cited by 594
Pantic, M., & Patras, I. (2006) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2005.859075 · Cited by 580
Bartlett, M. S., Littlewort, G., Fasel, I., & Movellan, J. R. (2003) · 2003 Conference on Computer Vision and Pattern Recognition Workshop · doi:10.1109/cvprw.2003.10057 · Cited by 573
Self-citationThe computer expression recognition toolbox (CERT)
Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., & Bartlett, M. (2011) · Face and Gesture 2011 · doi:10.1109/fg.2011.5771414 · Cited by 539
Self-citationThe Andes Physics Tutoring System: Lessons Learned
VanLehn, K. A., Lynch, C. F., Schulze, K. G., Shapiro, J. A., Shelby, R., Taylor, L., Treacy, D. J., Weinstein, A., & Wintersgill, M. C. (2005) · doi:10.5555/1434930.1434932 · Cited by 518
Automatic Recognition of Facial Actions in Spontaneous Expressions
Bartlett, M. S., Littlewort, G. C., Frank, M. G., Lainscsek, C., Fasel, I. R., & Movellan, J. R. (2006) · Journal of Multimedia · doi:10.4304/jmm.1.6.22-35 · Cited by 468
Self-citationBoredom in the Middle School Years: Blaming Schools versus Blaming Students
Larson, R. W., & Richards, M. H. (1991) · American Journal of Education · doi:10.1086/443992 · Cited by 454
Check & Connect: The importance of relationships for promoting engagement with school
Anderson, A. R., Christenson, S. L., Sinclair, M. F., & Lehr, C. A. (2004) · Journal of School Psychology · doi:10.1016/j.jsp.2004.01.002 · Cited by 395
Toward an Affect-Sensitive AutoTutor
D'Mello, S., Picard, R. W., & Graesser, A. (2007) · IEEE Intelligent Systems · doi:10.1109/mis.2007.79 · Cited by 386
Affect-aware tutors: recognising and responding to student affect
Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., & Picard, R. (2009) · International Journal of Learning Technology · doi:10.1504/ijlt.2009.028804 · Cited by 371
Dynamics of facial expression extracted automatically from video
Littlewort, G., Bartlett, M. S., Fasel, I., Susskind, J., & Movellan, J. (2006) · Image and Vision Computing · doi:10.1016/j.imavis.2005.09.011 · Cited by 358
Self-citationMultimodal affect recognition in learning environments
Kapoor, A., & Picard, R. W. (2005) · Proceedings of the 13th annual ACM international conference on Multimedia · doi:10.1145/1101149.1101300 · Cited by 339
D’Mello, S. K., & Graesser, A. (2010) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-010-9074-4 · Cited by 323
Toward Practical Smile Detection
Whitehill, J., Littlewort, G., Fasel, I., Bartlett, M., & Movellan, J. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.42 · Cited by 308
Self-citationA phenomenographic investigation of teacher conceptions of student engagement in learning
Harris, L. R. (2008) · The Australian Educational Researcher · doi:10.1007/bf03216875 · Cited by 246
Drowsy Driver Detection Through Facial Movement Analysis
Vural, E., Cetin, M., Ercil, A., Littlewort, G., Bartlett, M., & Movellan, J. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-75773-3_2 · Cited by 232
Self-citationAutomatic coding of facial expressions displayed during posed and genuine pain
Littlewort, G. C., Bartlett, M. S., & Lee, K. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.12.010 · Cited by 216
Prediction of subjective states from psychophysiology: A multivariate approach
Fairclough, S. H., & Venables, L. (2006) · Biological Psychology · doi:10.1016/j.biopsycho.2005.03.007 · Cited by 199
Engagement tracing: using response times to model student disengagement
Beck, J. E. (2005) · Cited by 176
A Time for Emoting: When Affect-Sensitivity Is and Isn’t Effective at Promoting Deep Learning
D’Mello, S. K., Lehman, B. A., Sullins, J., Daigle, R. P., Combs, R., Vogt, K. A., Perkins, L. A., & Graesser, A. C. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13388-6_29 · Cited by 138
D’Mello, S. K., Craig, S. D., Sullins, J., & Graesser, A. C. (2006) · doi:10.5555/1434940.1434942 · Cited by 136
Facial Features for Affective State Detection in Learning Environments
McDaniel, B., D’Mello, S. K., King, B. G., Chipman, P., Tapp, K. M., & Graesser, A. C. (2007) · Proceedings of the Annual Meeting of the Cognitive Science Society · Cited by 132
Repairing Disengagement With Non-Invasive Interventions
Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Meheranian, H., Fisher, D., Barto, A. G., Mahadevan, S., & Woolf, B. P. (2007) · Cited by 126
Makeig, S., Westerfield, M., Townsend, J., Jung, T., Courchesne, E., & B, T. J. S. A. (1999) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.1999.0469 · Cited by 109
Towards a Learning Companion that Recognizes Affect
Kapoor, A., Mota, S., & Picard, R. W. (2001) · Cited by 107
Multimethod assessment of affective experience and expression during deep learning
D’Mello, S. K., Craig, S. D., & Graesser, A. C. (2009) · International Journal of Learning Technology · doi:10.1504/ijlt.2009.028805 · Cited by 106
A dynamic mixture model to detect student motivation and proficiency
Johns, J., & Woolf, B. P. (2006) · Cited by 104
Automatic facial expression recognition for intelligent tutoring systems
Whitehill, J., Bartlett, M. S., & Movellan, J. R. (2008) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2008.4563182 · Cited by 87
Self-citationKnowledge Elicitation Methods for Affect Modelling in Education
Porayska‐Pomsta, K., Mavrikis, M., D’Mello, S. K., Conati, C., & Baker, R. S. (2013) · International Journal of Artificial Intelligence in Education · doi:10.3233/jai-130032 · Cited by 81
Viewing Student Affect and Learning through Classroom Observation and Physical Sensors
Dragon, T., Arroyo, I., Woolf, B. P., Burleson, W., el Kaliouby, R., & Eydgahi, H. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-69132-7_8 · Cited by 80
A Generic Forecast Verification Framework for Administrative Purposes
Mason, S. J., & Weigel, A. P. (2008) · Monthly Weather Review · doi:10.1175/2008mwr2553.1 · Cited by 78
Computer Expression Recognition Toolbox
Bartlett, M. S., Littlewort, G. C., Wu, T., & Movellan, J. R. (2008) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2008.4813406 · Cited by 56
Self-citationInterpreting Hand-Over-Face Gestures
Mahmoud, M., & Robinson, P. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_27 · Cited by 51
Multimodal analysis of the implicit affective channel in computer-mediated textual communication
Grafsgaard, J. F., Fulton, R. M., Boyer, K. E., Wiebe, E., & Lester, J. C. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388708 · Cited by 48
Multilayer Architectures for Facial Action Unit Recognition
Wu, T., Butko, N. J., Ruvolo, P., Whitehill, J., Bartlett, M. S., & Movellan, J. R. (2012) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2012.2195170 · Cited by 46
Self-citationGoldberg, B., Sottilare, R. A., Brawner, K., & Holden, H. K. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24600-5_57 · Cited by 44
Towards an Optimal Affect-Sensitive Instructional System of cognitive skills
Whitehill, J., Serpell, Z. N., Foster, A., Lin, Y.-C., Pearson, B., Bartlett, M. S., & Movellan, J. R. (2011) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2011.5981778 · Cited by 31
Self-citationAdapting to Multiple Affective States in Spoken Dialogue
Forbes-Riley, K., & Litman, D. J. (2012) · D-Scholarship@Pitt (University of Pittsburgh) · Cited by 27
Cited by
Application and theory gaps during the rise of artificial intelligence in education
Chen, X., Xie, H., Zou, D., & Hwang, G.-J. (2020) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2020.100002 · Cited by 1,003
Advanced, Analytic, Automated (AAA) Measurement of Engagement During Learning
D'Mello, S., Dieterle, E., & Duckworth, A. (2017) · Educational Psychologist · doi:10.1080/00461520.2017.1281747 · Cited by 350
Engagement detection in online learning: a review
Dewan, M. A. A., Murshed, M., & Lin, F. (2019) · Smart Learning Environments · doi:10.1186/s40561-018-0080-z · Cited by 350
Savchenko, A. V., Savchenko, L. V., & Makarov, I. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3188390 · Cited by 346
Automated Detection of Engagement Using Video-Based Estimation of Facial Expressions and Heart Rate
Monkaresi, H., Bosch, N., Calvo, R. A., & D'Mello, S. K. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2515084 · Cited by 329
Metadata
- Authors
- Whitehill, J., Serpell, Z., Lin, Y.-C., Foster, A., & Movellan, J. R.
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2014-01-01
- Volume
- 5
- Issue
- 1
- Pages
- 86-98
- ISSN
- 1949-3045 (ISSN-L)
- OpenAlex
- W2081112272
- Cited by
- 677
- References
- 53
- FWCI
- 28.83
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.