The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Automatic Detection of Learning-Centered Affective States in the Wild
Abstract
Affect detection is a key component in developing intelligent educational interfaces that are capable of responding to the affective needs of students. In this paper, computer vision and machine learning techniques were used to detect students' affect as they used an educational game designed to teach fundamental principles of Newtonian physics. Data were collected in the real-world environment of a school computer lab, which provides unique challenges for detection of affect from facial expressions (primary channel) and gross body movements (secondary channel) - up to thirty students at a time participated in the class, moving around, gesturing, and talking to each other. Results were cross validated at the student level to ensure generalization to new students. Classification was successful at levels above chance for off-task behavior (area under receiver operating characteristic curve or (AUC = .816) and each affective state including boredom (AUC =.610), confusion (.649), delight (.867), engagement (.679), and frustration (.631) as well as a five-way overall classification of affect (.655), despite the noisy nature of the data. Implications and prospects for affect-sensitive interfaces for educational software in classroom environments are discussed.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Methods for recognizing other affective states have also been proposed, namely cognitive states like boredom, confusion, delight, concentration and frustration [266], positive and negative affect from groups of people [267] or liking/not-linking of online media for predicting buying behaviour for marketing purposes [268].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [266]
In the citing paper
“Most approaches target primitive FEs of affect, but others recognize cognitive states [266], postive and negative affect from groups of people [267] and liking/not-linking of online media for predicting buying behaviour for marketing purposes [268].”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [266]
In the citing paper
“In naturalistic environments, most approaches target primitive FEs of affect. Methods for recognizing cognitive states [266], positive and negative affect from groups of people [267] or liking/not-linking of online media for predicting buying behaviour for marketing purposes [268] are also common.”
From Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications (2016) · cites this paper as [266]
Citations
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
An introduction to ROC analysis
Fawcett, T. (2006) · Pattern Recognition Letters · doi:10.1016/j.patrec.2005.10.010 · Cited by 21,879
Ekman, P., & Friesen, W. V. (2019) · PsycTESTS Dataset · doi:10.1037/t27734-000 · Cited by 6,019
Estimating attributes: Analysis and extensions of RELIEF
Kononenko, I. (1994) · Lecture notes in computer science · doi:10.1007/3-540-57868-4_57 · Cited by 3,062
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 4×
“Affect detection based on facial features has been the subject of considerable previous research (see [6,42] for reviews); albeit outside of learning contexts and mainly in laboratory settings (see exceptions discussed below).”
BackgroundInfluentialSelf-citationAllison, P. D. (1998) · Cited by 1,721
Dynamics of affective states during complex learning
D’Mello, S. K., & Graesser, A. C. (2011) · Learning and Instruction · doi:10.1016/j.learninstruc.2011.10.001 · Cited by 1,101
Self-citationWEKA: a machine learning workbench
Holmes, G., Donkin, A., & Witten, I. H. (2002) · Australian and New Zealand Intelligent Information Systems Conference (ANZIIS) · doi:10.1109/anziis.1994.396988 · Cited by 939
Facial signs of emotional experience.
Ekman, P., Freisen, W. V., & Ancoli, S. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077722 · Cited by 915
Facing Imbalanced Data--Recommendations for the Use of Performance Metrics
Jeni, L. A., Cohn, J. F., & de la Torre, F. N. (2013) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2013.47 · Cited by 828
Proceedings of the 33rd Annual Conference of the Cognitive Science Society
Dziemianko, M., Keller, F., & Coco, M. I. (2011) · Cited by 737
The Faces of Engagement: Automatic Recognition of Student Engagementfrom Facial Expressions
Whitehill, J., Serpell, Z. N., Lin, Y.-C., Foster, A., & Movellan, J. R. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2316163 · Cited by 734
Proceedings of the 7th International Conference on Educational Data Mining.
Stamper, J. C., Pardos, Z. A., Mavrikis, M., & McLaren, B. M. (2014) · Educational Data Mining · Cited by 667
Affect and learning: An exploratory look into the role of affect in learning with AutoTutor
Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004) · Journal of Educational Media · doi:10.1080/1358165042000283101 · Cited by 630
Automatic prediction of frustration
Kapoor, A., Burleson, W., & Picard, R. W. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.02.003 · Cited by 592
The computer expression recognition toolbox (CERT)
Littlewort, G. C., Whitehill, J., Wu, T., Fasel, I., Frank, M. G., Movellan, J. R., & Bartlett, M. S. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771414 · Cited by 571
Frenzel, A. C., Pekrun, R. H., & Goetz, T. (2007) · Learning and Instruction · doi:10.1016/j.learninstruc.2007.09.001 · Cited by 521
(2007) · Elsevier eBooks · doi:10.1016/b978-0-12-372545-5.x5000-x · Cited by 417
Ivon, A., Cooper David, G., Winslow, B., Park, W. B., Kasia, M., & Robert, C. (2009) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/978-1-60750-028-5-17 · Cited by 382
Multimodal affect recognition in learning environments
Kapoor, A., & Picard, R. W. (2005) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1101149.1101300 · Cited by 338
Journal of Artificial Intelligence Research (JAIR)
Minton, S. (1993) · Cited by 331
D’Mello, S. K. (2013) · Journal of Educational Psychology · doi:10.1037/a0032674 · Cited by 320
Self-citationAutomated Posture Analysis for Detecting Learner's Interest Level
Mota, S., & Picard, R. W. (2003) · IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2003.10047 · Cited by 315
Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors.
Lepper, M. R., Woolverton, M., Mumme, D. L., & Gurtner, J. (1993) · Cited by 293
Proceedings of the 29th Annual Cognitive Science Society
McNamara, D. S., & Trafton, J. G. (2007) · Cited by 268
McDuff, D., el Kaliouby, R., Senechal, T., Amr, M., Cohn, J. F., & Picard, R. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2013.130 · Cited by 234
Coherence between Emotion and Facial Expression: Evidence from Laboratory Experiments
Reisenzein, R., Studtmann, M., & Horstmann, G. (2013) · Emotion Review · doi:10.1177/1754073912457228 · Cited by 210
The half-life of cognitive-affective states during complex learning
D’Mello, S. K., & Graesser, A. C. (2011) · Cognition & Emotion · doi:10.1080/02699931.2011.613668 · Cited by 206
Self-citationEmotion recognition in the wild challenge 2013
Dhall, A., Goecke, R., Joshi, J., Wagner, M., & Gedeon, T. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531739 · Cited by 178
Exploring Temporal Patterns in Classifying Frustrated and Delighted Smiles
Hoque, M. E., McDuff, D., & Picard, R. W. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2012.11 · Cited by 165
Towards Sensor-Free Affect Detection in Cognitive Tutor Algebra.
Baker, R. S., Gowda, S. M., Wixon, M., Kalka, J., Wagner, A. Z., Salvi, A., Aleven, V. A. W. M. M., Kusbit, G. W., Ocumpaugh, J. L., & Rossi, L. M. (2012) · Cited by 154
Self-citationProceedings of the 33rd Annual Conference of the Cognitive Science Society
Dalzel-Job, S., & Oberlander, J. (2011) · Cited by 145
(1994) · Lecture notes in computer science · doi:10.1007/3-540-57868-4 · Cited by 136
Facial Features for Affective State Detection in Learning Environments
McDaniel, B., D’Mello, S. K., King, B. G., Chipman, P., Tapp, K. M., & Graesser, A. C. (2007) · eScholarship (California Digital Library) · Cited by 132
Self-citationAssessment and Learning of Qualitative Physics in Newton’s Playground
Ventura, M., Shute, V. J., & Kim, Y. J. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39112-5_63 · Cited by 110
Self-citationAutomatically Recognizing Facial Indicators of Frustration: A Learning-centric Analysis
Grafsgaard, J. F., Wiggins, J. B., Boyer, K. E., Wiebe, E., & Lester, J. C. (2013) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2013.33 · Cited by 102
Knowledge Elicitation Methods for Affect Modelling in Education
Porayska-Pomsta, K., Mavrikis, M., D'Mello, S. K., Conati, C., & Baker, R. (2013) · International Journal of Artificial Intelligence in Education · doi:10.3233/jai-130032 · Cited by 100
Self-citationHernandez, J., Hoque, M. E., Drevo, W., & Picard, R. W. (2012) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/2370216.2370264 · Cited by 99
The Dynamics of Affective Transitions in Simulation Problem-Solving Environments
Baker, R. S., Rodrigo, M. M. T., & Xolocotzin, U. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_58 · Cited by 74
Self-citationBosch, N., Chen, Y., & D’Mello, S. K. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07221-0_5 · Cited by 69
Self-citationAutomatic measurement of ad preferences from facial responses gathered over the Internet
McDuff, D., El Kaliouby, R., Senechal, T., Demirdjian, D., & Picard, R. (2014) · Image and Vision Computing · doi:10.1016/j.imavis.2014.01.004 · Cited by 52 · cited 1×
“This dataset has been used for detection of whether viewers liked the commercials (AUC = .82) and wanted to view them again (AUC = .79) [32].”
MethodImpact of action unit detection in automatic emotion recognition
Sénéchal, T., Bailly, K., & Prévost, L. (2012) · Pattern Analysis and Applications · doi:10.1007/s10044-012-0279-5 · Cited by 20
Ocumpaugh, J. L., Baker, R. S., Kamarainen, A. M., & Metcalf, S. (2014) · International Conference on User Modeling, Adaptation, and Personalization · Cited by 6
Self-citationDynamical Emotions: Bodily Dynamics of Affect during Problem Solving
D’Mello, S. K. (2011) · eScholarship (California Digital Library) · Cited by 4
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- 10.5555/1622407.1622416 · DOI
- Cook , D. L. The Hawthorne effect in educational research . Phi Delta Kappan , ( 1962 ), 116--122. Cook, D. L. The Hawthorne effect in educational research. Phi Delta Kappan, (1962), 116--122.
- D'Mello , S. , Blanchard , N. , Baker , R. , Ocumpaugh , J. , and Brawner , K . I feel your pain: A selective review of affect-sensitive instructional strategies. In R. Sottilare, A. Graesser, X. Hu and B. Goldberg, eds., Design Recommendations for Intelligent Tutoring Systems Volume 2 : Instructional Management . 2014 , 35 -- 48 . D'Mello, S., Blanchard, N., Baker, R., Ocumpaugh, J., and Brawner, K. I feel your pain: A selective review of affect-sensitive instructional strategies. In R. Sottilare, A. Graesser, X. Hu and B. Goldberg, eds., Design Recommendations for Intelligent Tutoring Systems Volume 2: Instructional Management. 2014, 35--48.
Cited by
Sorted by citations
Corneanu, C. A., Simon, M. O., Cohn, J. F., & Guerrero, S. E. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2515606 · Cited by 527
Stephanidis, C., Salvendy, G., Antona, M., Chen, J. Y. C., Dong, J., Duffy, V. G., Fang, X., Fidopiastis, C., Fragomeni, G., Fu, L. P., et al. (2019) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2019.1619259 · Cited by 524
Engagement detection in online learning: a review
Dewan, M. A. A., Murshed, M., & Lin, F. O. (2019) · Smart Learning Environments · doi:10.1186/s40561-018-0080-z · Cited by 390
Computer vision for assistive technologies
Leo, M., Medioni, G. G., Trivedi, M. M., Takeo Kanade, A., & Farinella, G. M. (2016) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2016.09.001 · Cited by 235
Multimodal data as a means to understand the learning experience
Giannakos, M. N., Sharma, K., Pappas, I. O., Kostakos, V., & Velloso, E. (2019) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.02.003 · Cited by 227
Metadata
- Authors
- Bosch, N., D'Mello, S., Baker, R., Ocumpaugh, J., Shute, V., Ventura, M., Wang, L., & Zhao, W.
- Affiliations
- University of Notre Dame (US)
- University of Notre Dame, Notre Dame, IN, USA
- Columbia University (US)
- Florida State University (US)
- Venue
- Proceedings of the 20th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2015-03-18
- Pages
- 379-388
- OpenAlex
- W2115417283
- Cited by
- 178
- References
- 45
- Citations by year
- 2026: 12 · 2025: 19 · 2024: 11 · 2023: 11 · 2022: 11 · 2021: 14 · 2020: 17 · 2019: 20 · 2018: 18 · 2017: 21 · 2016: 15 · 2015: 9
- FWCI
- 28.05
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Funders
- NSF (DRL 1235958)
- Bill & Melinda Gates Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- affective computing, affect recognition, game-based learning, facial expression recognition, body movement analysis, off-task behavior, boredom, confusion, engagement, frustration, delight, classroom climate
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
