The DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index.
A selective meta-analysis on the relative incidence of discrete affective states during learning with technology.
Abstract
The last decade has witnessed considerable interest in the investigation of the affective dimensions of learning and in the development of advanced learning technologies that automatically detect and respond to student affect. Identifying the affective states that students experience in technology-enhanced learning contexts is a fundamental question in this area. This article provides an initial attempt to answer this question with a selective meta-analysis of 24 studies that utilized a mixture of methodologies (online self-reports, online observations, emote-aloud protocols, cued recall) and affect judges (students themselves, untrained peers, trained judges) for fine-grained monitoring of 14 discrete affective states of 1,740 middle school, high school, college, and adult students in 5 countries. Affective states occurred over the course of interactions with a range of learning technologies, including intelligent tutoring systems, serious games, simulation environments, and simple computer interfaces. Standardized effect sizes of relative frequency, computed by comparing the proportional occurrence of each affective state to the other states in each study, were modeled with random-effects models. Engagement/flow was consistently found to be relatively frequent (d+ = 2.5), and contempt, anger, disgust, sadness, anxiety, delight, fear, and surprise were consistently infrequent, with d+ ranging from −6.5 to −0.78. Effects for boredom (d+ = 0.19), confusion (d+ = 0.12), curiosity (d+ = −0.10), happiness (d+ = −0.13), and frustration (d+ = −2.5) varied substantially across studies. Mixed-effects models indicated that the source of the affect judgments (self vs. observers) and the authenticity of the learning contexts (classroom vs. laboratory) accounted for greater heterogeneity than the use of advanced learning technologies and training time. Theoretical and applied implications of the findings are discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Conducting Meta-Analyses in R with the metafor Package
Viechtbauer, W. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i03 · Cited by 20,464
Flow: the psychology of optimal experience
(1990) · Choice Reviews Online · doi:10.5860/choice.28-0597 · Cited by 12,637
The origins of intelligence in children.
Piaget, J. (1952) · doi:10.1037/11494-000 · Cited by 10,687
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021) · doi:10.1002/9781119558378 · Cited by 9,799
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Mindset: The New Psychology of Success
(2006) · Choice Reviews Online · doi:10.5860/choice.44-2397 · Cited by 7,804
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
Csíkszentmihályi, M. (1975) · Cited by 5,690
The combination of estimates from different experiments
Cochran, W. G. (1954) · Biometrics · doi:10.2307/3001666 · Cited by 5,380 · cited 1×
“The homogeneity statistic (Q) (Cochran, 1954) was used to test if the true effect sizes were consistent (homogeneity) or varied across studies (heterogeneity), with significant Q’s being indicative of heterogeneity.”
MethodPekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002) · Educational Psychologist · doi:10.1207/s15326985ep3702_4 · Cited by 3,931
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
VanLEHN, K. (2011) · Educational Psychologist · doi:10.1080/00461520.2011.611369 · Cited by 2,323
The Cambridge handbook of multimedia learning
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5643 · Cited by 2,313
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893 · cited 2×
“…along with behavioral and physiological markers that have been extensively studied by researchers in the fields of nonverbal behavior and affective computing (see these reviews: Calvo & D‟Mello, 2010; D'Mello & Kory, 2012; Pantic & Rothkrantz, 2003; Valstar et al., in press; Zeng et al., 2009).”
BackgroundSelf-citationCordova, D. I., & Lepper, M. R. (1996) · Journal of Educational Psychology · doi:10.1037/0022-0663.88.4.715 · Cited by 1,568
We Feel, Therefore We Learn: The Relevance of Affective and Social Neuroscience to Education
Immordino‐Yang, M. H., & Damasio, A. (2007) · Mind, Brain, and Education · doi:10.1111/j.1751-228x.2007.00004.x · Cited by 1,463
Pekrun, R., Goetz, T., Daniels, L. M., Stupnisky, R. H., & Perry, R. P. (2010) · Journal of Educational Psychology · doi:10.1037/a0019243 · Cited by 1,182
Basic Emotions, Natural Kinds, Emotion Schemas, and a New Paradigm
Izard, C. E. (2007) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2007.00044.x · Cited by 1,110
Dynamics of affective states during complex learning
D’Mello, S., & Graesser, A. (2012) · Learning and Instruction · doi:10.1016/j.learninstruc.2011.10.001 · Cited by 1,087
Self-citationBaker, R. S. J. D., D'Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.12.003 · Cited by 936
Self-citationToward an affect-sensitive multimodal human-computer interaction
Pantic, M., & Rothkrantz, L. J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817122 · Cited by 791
Confusion can be beneficial for learning
D’Mello, S., Lehman, B., Pekrun, R., & Graesser, A. (2014) · Learning and Instruction · doi:10.1016/j.learninstruc.2012.05.003 · Cited by 777
Self-citationMind and Body: Psychology of Emotion and Stress
Mandler, G. (1984) · Medical Entomology and Zoology · Cited by 766
AutoTutor: an intelligent tutoring system with mixed-initiative dialogue.
Graesser, A. C., Chipman, P., Haynes, B. C., & Olney, A. (2005) · IEEE Transactions on Education · doi:10.1109/te.2005.856149 · Cited by 750
The Behavior of Tutoring Systems
VanLehn, K. (2006) · International Journal of Artificial Intelligence in Education · doi:10.3233/irg-2006-16(3)02 · Cited by 749
The power of statistical tests in meta-analysis
Hedges, L. V., & Pigott, T. D. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.3.203 · Cited by 716
Bong, M. (2001) · Journal of Educational Psychology · doi:10.1037//0022-0663.93.1.23 · Cited by 666
Reconceptalizing change in the cognitive construction of knowledge
Dole, J. A., & Sinatra, G. M. (1998) · Educational Psychologist · doi:10.1080/00461520.1998.9653294 · Cited by 665
Interest: A unique motivational variable
Hidi, S. (2006) · Educational Research Review · doi:10.1016/j.edurev.2006.09.001 · Cited by 643
Affect and learning: An exploratory look into the role of affect in learning with AutoTutor
Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004) · Journal of Educational Media · doi:10.1080/1358165042000283101 · Cited by 630
Automatic prediction of frustration
Kapoor, A., Burleson, W., & Picard, R. W. (2007) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2007.02.003 · Cited by 592
Levels of Analysis and the Organization of Affect
Rosenberg, E. L. (1998) · Review of General Psychology · doi:10.1037//1089-2680.2.3.247 · Cited by 563
Task values, achievement goals, and interest: An integrative analysis.
Hulleman, C. S., Durik, A. M., Schweigert, S. A., & Harackiewicz, J. M. (2008) · Journal of Educational Psychology · doi:10.1037/0022-0663.100.2.398 · Cited by 497
Re-conceptualizing Emotion and Motivation to Learn in Classroom Contexts
Meyer, D. K., & Turner, J. C. (2006) · Educational Psychology Review · doi:10.1007/s10648-006-9032-1 · Cited by 479
Off-task behavior in the cognitive tutor classroom: when students "game the system"
Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985741 · Cited by 453
Why Do Only Some Events Cause Learning During Human Tutoring?
VanLehn, K., Siler, S., Murray, C., Yamauchi, T., & Baggett, W. B. (2003) · Cognition and Instruction · doi:10.1207/s1532690xci2103_01 · Cited by 443
Between- and within-domain relations of students' academic emotions.
Goetz, T., Frenzel, A. C., Pekrun, R., Hall, N. C., & Lüdtke, O. (2007) · Journal of Educational Psychology · doi:10.1037/0022-0663.99.4.715 · Cited by 401
Goetz, T., Pekrun, R., Hall, N., & Haag, L. (2006) · British Journal of Educational Psychology · doi:10.1348/000709905x42860 · Cited by 390
Ivon, A., Cooper David, G., Winslow, B., Park, W. B., Kasia, M., & Robert, C. (2009) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/978-1-60750-028-5-17 · Cited by 382
Silvia, P. J. (2009) · Psychology of Aesthetics, Creativity, and the Arts · doi:10.1037/a0014632 · Cited by 340
Test Anxiety in Educational Contexts
Zeidner, M. (2007) · Emotion in Education · doi:10.1016/b978-012372545-5/50011-3 · Cited by 337
Gaze tutor: A gaze-reactive intelligent tutoring system
D'Mello, S., Olney, A., Williams, C., & Hays, P. (2012) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2012.01.004 · Cited by 335
Self-citationEmpirically Building and Evaluating a Probabilistic Model of User Affect
Conati, C., & Maclaren, H. (2009) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-009-9062-8 · Cited by 328
Berlyne, D. E. (1978) · Motivation and Emotion · doi:10.1007/bf00993037 · Cited by 326
Automatic detection of learner’s affect from conversational cues
D’Mello, S. K., Craig, S. D., Witherspoon, A., McDaniel, B., & Graesser, A. (2008) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-007-9037-6 · Cited by 317
Self-citationMeta-Analysis of the First Facial Expression Recognition Challenge
Valstar, M. F., Mehu, M., Jiang, B., Pantic, M., & Scherer, K. (2012) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2012.2200675 · Cited by 297
A New Paradigm for Intelligent Tutoring Systems: Example-Tracing Tutors
Aleven, V., McLaren, B. M., Sewall, J., & Koedinger, K. R. (2009) · doi:10.5555/1734243.1734245 · Cited by 277
Cognitive Computer Tutors: Solving the Two-Sigma Problem
Corbett, A. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44566-8_14 · Cited by 239
Academic boredom in under- and over-challenging situations
Acee, T. W., Kim, H., Kim, H. J., Kim, J.-I., Chu, H.-N. R., Kim, M., Cho, Y., & Wicker, F. W. (2010) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2009.08.002 · Cited by 237
Daschmann, E. C., Goetz, T., & Stupnisky, R. H. (2011) · British Journal of Educational Psychology · doi:10.1348/000709910x526038 · Cited by 228
Fear and anxiety: Overlaps and dissociations.
Öhman, A. (2008) · Cited by 218
Craig, S. D., D'Mello, S., Witherspoon, A., & Graesser, A. (2008) · Cognition & Emotion · doi:10.1080/02699930701516759 · Cited by 196
Self-citationThe half-life of cognitive-affective states during complex learning
D'Mello, S., & Graesser, A. (2011) · Cognition and Emotion · doi:10.1080/02699931.2011.613668 · Cited by 195
Self-citationDeci, E. L., & Ryan, R. M. (2002) · Improving Academic Achievement · doi:10.1016/b978-012064455-1/50007-5 · Cited by 183
Lepper, M. R., & Woolverton, M. (2002) · Improving Academic Achievement · doi:10.1016/b978-012064455-1/50010-5 · Cited by 180
New Perspectives on Affect and Learning Technologies
(2011) · Springer eBooks · doi:10.1007/978-1-4419-9625-1 · Cited by 173
Graesser, A. C., Lu, S., Olde, B. A., Cooper, E., & Whitten, S. N. (2005) · Memory & Cognition · doi:10.3758/bf03193225 · Cited by 160
Towards Sensor-Free Affect Detection in Cognitive Tutor Algebra.
Baker, R. S., Gowda, S. M., Wixon, M., Kalka, J., Wagner, A. Z., Salvi, A., Aleven, V. A. W. M. M., Kusbit, G. W., Ocumpaugh, J. L., & Rossi, L. M. (2012) · Educational Data Mining · Cited by 153
FAILURE TOLERANCE AND ACADEMIC RISK‐TAKING IN TEN‐ TO TWELVE‐YEAR‐OLD STUDENTS
Clifford, M. M. (1988) · British Journal of Educational Psychology · doi:10.1111/j.2044-8279.1988.tb00875.x · Cited by 151
A Time for Emoting: When Affect-Sensitivity Is and Isn’t Effective at Promoting Deep Learning
D’Mello, S. K., Lehman, B. A., Sullins, J., Daigle, R. P., Combs, R., Vogt, K. A., Perkins, L. A., & Graesser, A. C. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13388-6_29 · Cited by 138
Self-citationForbes-Riley, K., & Litman, D. J. (2011) · Speech Communication · doi:10.1016/j.specom.2011.02.006 · Cited by 133
Facial Features for Affective State Detection in Learning Environments
McDaniel, B., D’Mello, S. K., King, B. G., Chipman, P., Tapp, K. M., & Graesser, A. C. (2007) · Proceedings of the Annual Meeting of the Cognitive Science Society · Cited by 132
Self-citationWoolf, B. P., Arroyo, I., Müldner, K., Burleson, W., Cooper, D. G., Dolan, R., & Christopherson, R. M. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13388-6_37 · Cited by 124
The Role of Motivational Beliefs in Conceptual Change
Linnenbrink, E. A., & Pintrich, P. R. (2006) · Kluwer Academic Publishers eBooks · doi:10.1007/0-306-47637-1_6 · Cited by 123
Detection of Emotions during Learning with AutoTutor
Chipman, P., D’Mello, S. K., Gholson, B., Graesser, A. C., McDaniel, B., & Witherspoon, A. M. (2006) · Proceedings of the Annual Meeting of the Cognitive Science Society · Cited by 120
Self-citationLitman, D. J., & Forbes-Riley, K. (2005) · Speech Communication · doi:10.1016/j.specom.2005.09.008 · Cited by 118
Modeling Learner Affect with Theoretically Grounded Dynamic Bayesian Networks
Sabourin, J. L., Mott, B., & Lester, J. C. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24600-5_32 · Cited by 110
Towards Systems That Care: A Conceptual Framework based on Motivation, Metacognition and Affect
du Boulay, B., Avramides, K., Luckin, R., Martínez-Mirón, E. A., Rebolledo‐Mendez, G., & Carr, A. (2010) · International Journal of Artificial Intelligence in Education · doi:10.3233/jai-2010-0007 · Cited by 83
Adapting to Student Uncertainty Improves Tutoring Dialogues
Forbes-Riley, K., & Litman, D. J. (2009) · Frontiers in artificial intelligence and applications · doi:10.3233/978-1-60750-028-5-33 · Cited by 79
The Dynamics of Affective Transitions in Simulation Problem-Solving Environments
Baker, R. S., Rodrigo, M. M. T., & Xolocotzin, U. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_58 · Cited by 76
Emotions and Learning with AutoTutor
Graesser, A. C., Chipman, P., King, B. G., McDaniel, B., & D’Mello, S. K. (2007) · Cited by 75
Self-citationEvaluation of animalwatch: An intelligent tutoring system for arithmetic and fractions
Beal, C. R., Arroyo, I., Cohen, P. R., & Woolf, B. P. (2010) · Cited by 74
Acted vs. natural frustration and delight: Many people smile in natural frustration
Hoque, M. E., & Picard, R. W. (2011) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2011.5771425 · Cited by 70
Experienced emotions, emotion regulation and student activity in a web-based learning environment
Vuorela, M., & Nummenmaa, L. (2004) · European Journal of Psychology of Education · doi:10.1007/bf03173219 · Cited by 61
Smile When You Read This, Whether You Like It or Not: Conceptual Challenges to Affect Detection
Kappas, A. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.6 · Cited by 57
The Impact of System Feedback on Learners’ Affective and Physiological States
Pour, P. A., Hussain, M. S., AlZoubi, O., D’Mello, S. K., & Calvo, R. A. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13388-6_31 · Cited by 53
Self-citationEmotions while writing about emotional and non-emotional topics
D’Mello, S. K., & Mills, C. (2013) · Motivation and Emotion · doi:10.1007/s11031-013-9358-1 · Cited by 46
Self-citationWhen Does Disengagement Correlate with Performance in Spoken Dialog Computer Tutoring?
Forbes-Riley, K., & Litman, D. J. (2013) · International Journal of Artificial Intelligence in Education · doi:10.3233/jai-130028 · Cited by 39
The Dynamics between Student Affect and Behavior Occurring Outside of Educational Software
Baker, R. S., Moore, G. R., Wagner, A. Z., Kalka, J., Salvi, A., Karabinos, M., Ashe, C. A., & Yaron, D. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24600-5_5 · Cited by 35
Exploring the Relationship between Learner EEG Mental Engagement and Affect
Chaouachi, M., & Frasson, C. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13437-1_48 · Cited by 35
Lehman, B. A., D’Mello, S. K., Strain, A. C., Gross, M., Dobbins, A., Wallace, P. S., Millis, K., & Graesser, A. C. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21869-9_24 · Cited by 30
Self-citationRodrigo, M. M. T., & Baker, R. S. (2011) · Springer eBooks · doi:10.1007/978-1-4419-9625-1_14 · Cited by 29
Emotion Regulation during Learning
Strain, A. C., & D’Mello, S. K. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21869-9_103 · Cited by 27
Self-citationHarley, J. M., Bouchet, F., & Azevedo, R. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-30950-2_5 · Cited by 27
Natural Affect Data: Collection and Annotation
Afzal, S., & Robinson, P. (2011) · Springer eBooks · doi:10.1007/978-1-4419-9625-1_5 · Cited by 26
Cited by
A Review and Meta-Analysis of Multimodal Affect Detection Systems
D'mello, S. K., & Kory, J. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 628
Self-citationTeacher emotions in the classroom and their implications for students
Frenzel, A. C., Daniels, L., & Burić, I. (2021) · Educational Psychologist · doi:10.1080/00461520.2021.1985501 · Cited by 470
Learner Engagement in Blended Learning Environments: A Conceptual Framework
Halverson, L. R., & Graham, C. R. (2019) · Online Learning · doi:10.24059/olj.v23i2.1481 · Cited by 437
Loderer, K., Pekrun, R., & Lester, J. C. (2020) · Learning and Instruction · doi:10.1016/j.learninstruc.2018.08.002 · Cited by 377
Measuring emotions during epistemic activities: the Epistemically-Related Emotion Scales
Pekrun, R., Vogl, E., Muis, K. R., & Sinatra, G. M. (2017) · Cognition and Emotion · doi:10.1080/02699931.2016.1204989 · Cited by 373
Metadata
- Authors
- D'Mello, S.
- Venue
- Journal of Educational Psychology
- Published
- 2013-11
- Volume
- 105
- Issue
- 4
- Pages
- 1082-1099
- ISSN
- 0022-0663 (ISSN-L)
- 1939-2176
- OpenAlex
- W2171618137
- Cited by
- 300
- References
- 86
- FWCI
- 19.88
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.