Heart rates, facial expressions and self-reports: a multimodal longitudinal approach of learners' emotions in the foreign language classroom
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Emotions in educational settings are often studied through self-reports or lab experiments, limiting insights into their real-world dynamics. This study examines learner emotions in authentic foreign language classrooms using a multimodal longitudinal approach. Over 16 consecutive sessions, we collected heart rate (HR) signals, emotional facial expressions (EFE), classroom observations, and self-reports on enjoyment, anxiety, and boredom to capture both physiological and self-perceived emotional responses. Rather than aggregating data across students, we focused on individualized emotional patterns to understand variations in emotional experiences. Each dataset included extensive video recordings, continuous HR monitoring, detailed observational notes, and post-session questionnaires, providing a high-resolution picture of emotional dynamics. Using unsupervised clustering techniques, we identified key emotional episodes—peaks and drops in physiological arousal (heart rate variation) and facial expression—relative to individual emotional baselines. These moments were cross-referenced with classroom observations and self-reports for validation. Findings highlight moments of positive emotional contagion during peer interactions, emphasizing the social dimension of language learning. This multimodal approach captures the interplay of physiological, behavioral, and subjective responses, offering a scalable method for studying classroom emotions. Methodologically, it demonstrates how multimodal analytics can uncover transient emotional states in real-world settings, while practically informing adaptive teaching strategies, such as leveraging peer interactions to enhance engagement or reduce anxiety. By integrating physiological, behavioral, and subjective data, this study provides a comprehensive framework for understanding the affective dimensions of learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Foreign Language Classroom Anxiety
HORWITZ, E. K., HORWITZ, M. B., & COPE, J. (1986) · The Modern Language Journal · doi:10.1111/j.1540-4781.1986.tb05256.x · Cited by 5,697 · cited 2×
“Adopting a dynamic approach, Gregersen et al. (2014) conducted a multimodal study on three high-anxiety and three low-anxiety learners of Spanish FL [based on scores on Horwitz et al. (1986) FL Classroom Anxiety Scale].”
Autonomic nervous system activity in emotion: a review
Kreibig, S. D. (2010) · Biological Psychology · doi:10.1016/j.biopsycho.2010.03.010 · Cited by 3,188 · cited 4×
“For the EFE analysis, in contrast to HR signal processing, where variable T chunk durations are employed based on flexible guidelines (Kreibig, 2010), we focused on frame-level recognition.”
InfluentialThe two faces of Janus? Anxiety and enjoyment in the foreign language classroom
Dewaele, J.-M., & MacIntyre, P. D. (2014) · Studies in Second Language Learning and Teaching · doi:10.14746/ssllt.2014.4.2.5 · Cited by 1,757
Self-citationA review of emotion recognition using physiological signals
Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., & Yang, X. (2018) · Sensors · doi:10.3390/s18072074 · Cited by 923 · cited 3×
“(Wang et al., 2022; Shu et al., 2018; Poria et al., 2017; Wu, 2023) have highlighted HR data as a reliable measure of physiological states, relying solely on one modality may overlook important behavioral cues that provide additional insight into engagement.”
Emotions that facilitate language learning: the positive-broadening power of the imagination
MacIntyre, P., & Gregersen, T. (2012) · Studies in Second Language Learning and Teaching · doi:10.14746/ssllt.2012.2.2.4 · Cited by 843
Challenges in representation learning: A report on three machine learning contests
Goodfellow, I. J., Erhan, D., Luc Carrier, P., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.005 · Cited by 697
Dewaele, J.-M., & Li, C. (2021) · Language Teaching Research · doi:10.1177/13621688211014538 · Cited by 532
Self-citationThe Motion of Emotion: Idiodynamic Case Studies of Learners' Foreign Language Anxiety
Gregersen, T., Macintyre, P. D., & Meza, M. D. (2014) · The Modern Language Journal · doi:10.1111/modl.12084 · Cited by 335
Botes, E., Dewaele, J.-M., & Greiff, S. (2020) · Journal for the Psychology of Language Learning · doi:10.52598/jpll/2/1/3 · Cited by 333
Self-citation
Research works printed but not linked: 54
- Agrawal, 2022, The temporal dynamics of opportunity costs: a normative account of cognitive fatigue and boredom, Psychol. Rev, 129, 564, 10.1037/rev0000309 · DOI
- Barrett, 2017, How Emotions Are Made: The Secret Life of the Brain
- Bielak, 2024, Emotions and emotion regulation in l2 classroom speaking tasks: a mixed-methods study combining the idiodynamic and quantitative perspectives, Mod. Lang. J, 108, 688, 10.1111/modl.12950 · DOI
- Botes, 2021, The development and validation of the short form of the foreign language enjoyment scale, Mod. Lang. J, 105, 858, 10.1111/modl.12741 · DOI · Paper page
- Botes, Validating the short-form foreign language classroom anxiety scale, Appl. Ling, 43, 1006, 10.1093/applin/amac018 · DOI · Paper page
- Boudreau, 2018, Enjoyment and anxiety in second language communication: an idiodynamic approach, Stud. Second Lang. Learn. Teach, 8, 149, 10.14746/ssllt.2018.8.1.7 · DOI · Paper page
- Csikszentmihalyi, 1990, Flow: The Psychology of Optimal Experience
- De Bot, 2007, A dynamic systems theory approach to second language acquisition, Biling.: Lang. Cogn, 10, 7, 10.1017/S1366728906002732 · DOI · Paper page
- Dewaele, 2023, “Collecting and analyzing individual differences data in foreign language learning,”, Current Approaches in Second Language Acquisition Research: A Practical Guide, 10.1002/9781394259670.ch10 · DOI
- Dewaele, 2022, Do flow, enjoyment and anxiety emerge equally in English foreign language classrooms as in other foreign language classrooms?, Rev. Brasil. Ling. Aplicada, 22, 156, 10.1590/1984-6398202218487 · DOI
- Dewaele, 2019, The relationship between incommensurable emotions and willingness to communicate in English as a foreign language: a multiple case study, Innov. Lang. Learn. Teach, 15, 66, 10.1080/17501229.2019.1675667 · DOI · Paper page
- D'Mello, 2012, Gaze tutor: a gaze-reactive intelligent tutoring system, Int. J. Hum. Comput. Stud, 70, 377, 10.1016/j.ijhcs.2012.01.004 · DOI · Paper page
- Dörnyei, 2007, Research Methods in Applied Linguistics, 335, 10.4000/asp.294 · doi:10.4000/asp.294
- Ekman, 1992, An argument for basic emotions, Cogn. Emot, 6, 169, 10.1080/02699939208411068 · DOI · Paper page
- Elahi Shirvan, 2020, The dynamics of foreign language enjoyment: an ecological momentary assessment, Front. Psychol, 11, 1391, 10.3389/fpsyg.2020.01391 · DOI · Paper page
- Gendron, 2018, La perception des émotions: une synchronie conceptuelle, Sensibilités, 2, 70, 10.3917/sensi.005.0070 · doi:10.3917/sensi.005.0070
- Gobin, 2021, “Les émotions,”, Emotions et Apprentissages, 10.3917/dunod.gobin.2021.01.0017 · DOI
- Hiver, 2022, Complex dynamic systems theory in language learning: a scoping review of 25 years of research, Stud. Second Lang. Acquisit, 44, 913, 10.1017/S0272263121000553 · DOI
- Hoemann, 2023, Multimodal, idiographic ambulatory sensing will transform our understanding of emotion, Affect. Sci, 4, 480, 10.1007/s42761-023-00206-0 · DOI
- Horwitz, 2017, On the misreading of horwitz, horwitz, and cope (1986) and the need to balance anxiety research and the experiences of anxious language learners, New Insights Lang. Anxiety, 31, 47, 10.2307/jj.22730706.6 · doi:10.2307/jj.22730706.6 · Paper page
- Iyriboz, 1991, Accuracy of pulse oximeters in estimating heart rate at rest and during exercise, Br. J. Sports Med, 25, 162, 10.1136/bjsm.25.3.162 · DOI · Paper page
- Kim, 2008, Emotion recognition based on physiological changes in music listening, IEEE Trans. Pattern Anal. Mach. Intell, 30, 2067, 10.1109/TPAMI.2008.26 · DOI · Paper page
- Lambert, 2023, “The role of the learner in task-based language teaching,”, 1, 10.4324/9781003227267-2 · doi:10.4324/9781003227267-2
- Lazarus, 1991, Emotion and Adaptation, 10.1093/oso/9780195069945.001.0001 · DOI · Paper page
- Li, 2021, A control-value theory approach to boredom in English classes among university students in China, Mod. Lang. J, 105, 317, 10.1111/modl.12693 · DOI · Paper page
- Li, 2024, “Understanding, measuring, and differentiating task enjoyment from foreign language enjoyment,”, Individual Differences and Task-Based Language Teaching, 10.1075/tblt.16.03lic · DOI
- Li, 2021, How classroom environment and general grit predict foreign language classroom anxiety of Chinese EFL students, J. Psychol. Lang. Learn, 3, 86, 10.52598/jpll/3/2/6 · DOI
- Li, 2023, Foreign language learning boredom: conceptualization and measurement, Appl. Ling. Rev, 14, 223, 10.1515/applirev-2020-0124 · DOI · Paper page
- Li, 2022, Classroom environment and willingness to communicate in English: the mediating role of emotions experienced by university students in China, Lang. Teach. Res, 29, 2140, 10.1177/13621688221111623 · DOI · Paper page
- Li, 2020, Deep facial expression recognition: a survey, IEEE Trans. Affect. Comput, 13, 1195, 10.1109/TAFFC.2020.2981446 · DOI · Paper page
- MacIntyre, 2022, The idiodynamic method: a practical guide for researchers, Res. Methods Appl. Ling, 1, 100007, 10.1016/j.rmal.2022.100007 · DOI
- MacIntyre, 2023, “The idiodynamic method,”, 125, 10.4324/9781003227267-11 · doi:10.4324/9781003227267-11
- MacIntyre, 2019, The emotional underpinnings of Gardner's attitudes and motivation test battery, Contemp. Lang. Motiv. Theory, 60, 57, 10.2307/jj.22730665.10 · DOI
- Mackey, 2023, Current Approaches in Second Language Acquisition Research: A Practical Guide, 10.1002/9781394259670 · DOI
- Mera, 2004, “Emotion analyzing method using physiological state,”, 195
- Mollahosseini, 2017, Affectnet: a database for facial expression, valence, and arousal computing in the wild, IEEE Trans. Affect. Comput, 10, 18, 10.1109/TAFFC.2017.2740923 · DOI · Paper page
- Moskowitz, 2021, Is teacher happiness contagious? A study of the link between perceptions of language teacher happiness and student attitudes, Innov. Lang. Learn. Teach, 15, 117, 10.1080/17501229.2019.1707205 · DOI · Paper page
- Nozawa, 2021, Prefrontal inter-brain synchronization reflects convergence and divergence of flow dynamics in collaborative learning: a pilot study, Front. Neuroergon, 2, 686596, 10.3389/fnrgo.2021.686596 · DOI
- Pawlak, 2024, “Teachers reflecting on boredom in the language classroom,”, 10.3138/9781800504233 · doi:10.3138/9781800504233
- Petiot, 2022, Les émotions en contexte scolaire, Admin, 176, 9, 10.3917/admed.176.0009 · doi:10.3917/admed.176.0009
- Poria, 2017, A review of affective computing: from unimodal analysis to multimodal fusion, Inf. Fusion, 37, 98, 10.1016/j.inffus.2017.02.003 · DOI · Paper page
- Rousseeuw, 1987, Silhouettes: a graphical aid to the interpretation and validation of cluster analysis, J. Comput. Appl. Math, 20, 53, 10.1016/0377-0427(87)90125-7 · DOI · Paper page
- Sauvage, 2022, Humanités numériques et pensée complexe, LHUMAINE., 10.34745/numerev_2001 · doi:10.34745/numerev_2001
- Schroff, 2015, “Facenet: a unified embedding for face recognition and clustering,”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 10.1109/CVPR.2015.7298682 · DOI · Paper page
- Shu, 2020, Wearable emotion recognition using heart rate data from a smart bracelet, Sensors, 20, 718, 10.3390/s20030718 · DOI · Paper page
- Talebzadeh, 2020, Dynamics and mechanisms of foreign language enjoyment contagion, Innov. Lang. Learn. Teach, 14, 399, 10.1080/17501229.2019.1614184 · DOI
- Tonguç, 2020, Automatic recognition of student emotions from facial expressions during a lecture, Comput. Educ, 148, 103797, 10.1016/j.compedu.2019.103797 · DOI
- Wang, 2023, Connecting foreign language enjoyment and English proficiency levels: the mediating role of l2 motivation, Front. Psychol, 14, 1054657, 10.3389/fpsyg.2023.1054657 · DOI
- Wang, 2022, A systematic review on affective computing: emotion models, databases, and recent advances, Inf. Fusion, 83, 19, 10.1016/j.inffus.2022.03.009 · DOI · Paper page
- Waninge, 2014, Motivational dynamics in language learning: change, stability, and context, Mod. Lang. J, 98, 704, 10.1111/modl.12118 · doi:10.1111/modl.12118 · Paper page
- Wu, 2023, Multimodal emotion recognition from physiological signals and facial expressions
- Wukitsch, 1988, Pulse oximetry: analysis of theory, technology, and practice, J. Clin. Monit, 4, 290, 10.1007/BF01617328 · DOI · Paper page
- Yu, 2022, A review of foreign language learners' emotions, Front. Psychol, 12, 827104, 10.3389/fpsyg.2021.827104 · DOI
- Yu, 2023, Real-time EEG-based emotion recognition, Sensors, 23, 7853, 10.3390/s23187853 · DOI
Metadata
- Authors
- Guedat-Bittighoffer, D., Moufidi, A., Dewaele, J.-M., Rousseau, D., Voyneau, H., & Rasti, P.
- Venue
- Frontiers in Artificial Intelligence
- Publisher
- Frontiers Media SA
- Published
- 2025-10-01
- Volume
- 8
- ISSN
- 2624-8212
- Cited by
- 1
- References
- 14
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Heart rates, facial expressions and self-reports: a multimodal longitudinal approach of learners' emotions in the foreign language classroom. https://app.cochat.ai/papers/10.3389/frai.2025.1604110
- The paper
- Guedat-Bittighoffer, D., Moufidi, A., Dewaele, J.-M., Rousseau, D., Voyneau, H., & Rasti, P. (2025). Heart rates, facial expressions and self-reports: a multimodal longitudinal approach of learners' emotions in the foreign language classroom. Frontiers in Artificial Intelligence. https://doi.org/10.3389/frai.2025.1604110
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.