Predicting Speaker Head Nods and the Effects of Affective Information
Abstract
During face-to-face conversation, our body is continually in motion, displaying various head, gesture, and posture movements. Based on findings describing the communicative functions served by these nonverbal behaviors, many virtual agent systems have modeled them to make the virtual agent look more effective and believable. One channel of nonverbal behaviors that has received less attention is head movements, despite the important functions served by them. The goal for this work is to build a domain-independent model of speaker's head movements that could be used to generate head movements for virtual agents. In this paper, we present a machine learning approach for learning models of head movements by focusing on when speaker head nods should occur, and conduct evaluation studies that compare the nods generated by this work to our previous approach of using handcrafted rules . To learn patterns of speaker head nods, we use a gesture corpus and rely on the linguistic and affective features of the utterance. We describe the feature selection process and training process for learning hidden Markov models and compare the results of the learned models under varying conditions. The results show that we can predict speaker head nods with high precision (.84) and recall (.89) rates, even without a deep representation of the surface text and that using affective information can help improve the prediction of the head nods (precision: .89, recall: .90). The evaluation study shows that the nods generated by the machine learning approach are perceived to be more natural in terms of nod timing than the nods generated by the rule-based approach.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the field of conversational agents, head gesture generation and evaluation has mainly focussed on head movements as co-speech speaker-in contrast to listener-behavior.
“In the field of conversational agents, head gesture generation and evaluation, has, however, mainly focussed on head movements (including nods) as co-speech speaker-in contrast to listenerbehavior (e.g., Lee and Marsella, 2010; Ding et al., 2013).”
From Modeling Feedback in Interaction With Conversational Agents—A Review (2022) · cites this paper as Lee and Marsella, 2010
Cites
A tutorial on hidden Markov models and selected applications in speech recognition
Rabiner, L. R. (1989) · Proceedings of the IEEE · doi:10.1109/5.18626 · Cited by 22,743 · see citing works
Nonverbal communication in human interaction
Knapp, M. L. (1972) · Cited by 2,209 · see citing works
(1999) · Cited by 1,521 · see citing works
Hidden Markov Models for Speech Recognition
Juang, B.-H., & Rabiner, L. R. (1991) · Technometrics · doi:10.2307/1268779 · Cited by 1,069 · see citing works
The nature of rapport and its nonverbal correlates
Tickle-Degnen, L., & Rosenthal, R. (1990) · Psychological Inquiry · doi:10.1207/s15327965pli0104_1 · Cited by 960 · see citing works
Charniak, E. (1994) · Medical Entomology and Zoology · Cited by 777 · see citing works
BEAT: the Behavior Expression Animation Toolkit
Cassell, J., Vilhjálmsson, H. H., & Bickmore, T. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383315 · Cited by 739 · see citing works
Parametric hidden Markov models for gesture recognition
Wilson, A. D., & Bobick, A. (1999) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.790429 · Cited by 574 · see citing works
Handbook of interpersonal communication
Pettegrew, L. S. (1987) · Social Science & Medicine · doi:10.1016/0277-9536(87)90318-2 · Cited by 524 · see citing works
Linguistic functions of head movements in the context of speech
McClave, E. Z. (2000) · Journal of Pragmatics · doi:10.1016/s0378-2166(99)00079-x · Cited by 500 · see citing works
Visual Prosody and Speech Intelligibility: Head Movement Improves Auditory Speech Perception
Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., & Vatikiotis-Bateson, E. (2004) · Psychological Science · doi:10.1111/j.0963-7214.2004.01502010.x · Cited by 482 · see citing works
Embodied conversational interface agents
Cassell, J. (2000) · Communications of the ACM · doi:10.1145/332051.332075 · Cited by 388 · see citing works
Statistical Language Learning.
SW, & Charniak, E. (1994) · Journal of the American Statistical Association · doi:10.2307/2291039 · Cited by 367 · see citing works
Unleashing the killer corpus: experiences in creating the multi-everything AMI Meeting Corpus
Carletta, J. (2007) · Computers and the Humanities · doi:10.1007/s10579-007-9040-x · Cited by 362 · see citing works
American-Japanese Cultural Differences in the Recognition of Universal Facial Expressions
Matsumoto, D. (1992) · Journal of Cross-Cultural Psychology · doi:10.1177/0022022192231005 · Cited by 289 · see citing works
Nonverbal Behavior Generator for Embodied Conversational Agents
Lee, J., & Marsella, S. (2006) · Lecture notes in computer science · doi:10.1007/11821830_20 · Cited by 240 · see citing works
Self-citationSwartout, W., Gratch, J. S., Hill, R. W., Hovy, E. H., Marsella, S., Rickel, J. W., & Traum, D. R. (2006) · doi:10.1609/aimag.v27i2.1883 · Cited by 222 · see citing works
Self-citationToward a new generation of virtual humans for interactive experiences
Rickel, J. W., Marsella, S., Gratch, J. S., Hill, R., Traum, D. R., & Swartout, W. (2002) · IEEE Intelligent Systems · doi:10.1109/mis.2002.1024750 · Cited by 221 · see citing works
Self-citationThe Many Faces of a Neutral Face: Head Tilt and Perception of Dominance and Emotion
Mignault, A., & Chaudhuri, A. (2003) · Journal of Nonverbal Behavior · doi:10.1023/a:1023914509763 · Cited by 204 · see citing works
Kendon, A. (2002) · Gesture · doi:10.1075/gest.2.2.03ken · Cited by 199 · see citing works
Visual prosody: facial movements accompanying speech
Graf, H. P., Cosatto, E., Strom, V., & Huang, F. J. (2003) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2002.1004186 · Cited by 199 · see citing works
Rigid Head Motion in Expressive Speech Animation: Analysis and Synthesis
Busso, C., Deng, Z., Grimm, M., Neumann, U., & Narayanan, S. S. (2007) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2006.885910 · Cited by 181 · see citing works
Textual Affect Sensing for Sociable and Expressive Online Communication
Neviarouskaya, A., Prendinger, H., & Ishizuka, M. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_20 · Cited by 137 · see citing works
Interactional functions of a nonverbal sign Head movement in japanese dyadic casual conversation
Maynard, S. K. (1987) · Journal of Pragmatics · doi:10.1016/0378-2166(87)90181-0 · Cited by 131 · see citing works
Natural Behavior of a Listening Agent
Maatman, R. M., Gratch, J. S., & Marsella, S. (2005) · Lecture notes in computer science · doi:10.1007/11550617_3 · Cited by 129 · see citing works
Self-citationVirtual Patients for Clinical Therapist Skills Training
Kenny, P. G., Parsons, T. D., Gratch, J. S., Leuski, A., & Rizzo, A. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74997-4_19 · Cited by 122 · see citing works
PEDAGOGICALLY STRUCTURED GAME-BASED TRAINING: DEVELOPMENT OF THE ELECT BILAT SIMULATION
Hill, R. W., Dixon, M. W., Forbell, E., Kim, J., Hart, J., Belanich, J., Lane, H. C., & Core, M. G. (2006) · doi:10.13140/2.1.2902.7520 · Cited by 110 · see citing works
Predicting Listener Backchannels: A Probabilistic Multimodal Approach
Morency, L., de Kok, I., & Gratch, J. S. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85483-8_18 · Cited by 109 · see citing works
More Than Just Another Pretty Face: Embodied Conversational Interface Agents
Cassell, J. (1999) · Cited by 103 · see citing works
Kipp, M., Neff, M., Kipp, K. H., & Albrecht, I. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74997-4_2 · Cited by 96 · see citing works
A shallow model of backchannel continuers in spoken dialogue
Cathcart, N., Carletta, J., & Klein, E. (2003) · Proceedings of the tenth conference on European chapter of the Association for Computational Linguistics - EACL '03 · doi:10.3115/1067807.1067816 · Cited by 88 · see citing works
Challenges ahead: head movements and other social acts during conversations
Heylen, D. (2005) · Cited by 56 · see citing works
Corpus-based generation of head and eyebrow motion for an embodied conversational agent
Foster, M. E., & Oberlander, J. (2007) · Computers and the Humanities · doi:10.1007/s10579-007-9055-3 · Cited by 51 · see citing works
A Spoken Dialog System for Chat-Like Conversations Considering Response Timing
Nishimura, R., Kitaoka, N., & Nakagawa, S. (2007) · Springer eBooks · doi:10.1007/978-3-540-74628-7_77 · Cited by 39 · see citing works
Hassan: A Virtual Human for Tactical Questioning
Traum, D. R., Roque, A., Leuski, A., Georgiou, P., Gerten, J., Martinovski, B., Narayanan, S. S., Robinson, S. M., & Vaswani, A. (2007) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2007.sigdial-1.15 · Cited by 38 · see citing works
Virtual gestures: embodiment and nonverbal behavior in computer-mediated communication
Bente, G., & Krämer, N. C. (2011) · Cambridge University Press eBooks · doi:10.1017/cbo9780511977589.010 · Cited by 30 · see citing works
Glances, glares, and glowering: how should a virtual human express emotion through gaze?
Lance, B. J., & Marsella, S. (2009) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-009-9097-6 · Cited by 27 · see citing works
Self-citationEric: a generic rule-based framework for an affective embodied commentary agent
Strauss, M. (2008) · Cited by 22 · see citing works
Learning a model of speaker head nods using gesture corpora
Lee, J., & Marsella, S. (2009) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/fbzs9841 · Cited by 21 · see citing works
Self-citationAutomated generation of non-verbal behavior for virtual embodied characters
Breitfuß, W., Prendinger, H., & Ishizuka, M. (2007) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1322192.1322247 · Cited by 19 · see citing works
Learning models of speaker head nods with affective information
Lee, J., Prendinger, H., Neviarouskaya, A., & Marsella, S. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349543 · Cited by 17 · see citing works
Self-citationChallenges Ahead. Head movements and other social acts in conversations
Heylen, D., Halle, L., Wallis, P., Woods, S., Marsella, S., Pélachaud, C., & Heylen, D. K. J. (2005) · University of Twente Research Information · Cited by 16 · see citing works
Self-citationTrackside DEIRA: a dynamic engaging intelligent reporter agent
Knoppel, F. L. A., Tigelaar, A. S., Bos, D. O., Alofs, T., & Ruttkay, Z. (2008) · Adaptive Agents and Multi-Agents Systems · doi:10.5555/1402383.1402404 · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
MFDNet: Collaborative Poses Perception and Matrix Fisher Distribution for Head Pose Estimation
Liu, H., Fang, S., Zhang, Z., Li, D., Lin, K., & Wang, J. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3081873 · Cited by 254 · see citing works
Virtual character performance from speech
Marsella, S., Xu, Y., Lhommet, M., Feng, A., Scherer, S., & Shapiro, A. (2013) · Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation · doi:10.1145/2485895.2485900 · Cited by 168 · see citing works
Self-citationNew Modality: Emoji Challenges in Prediction, Anticipation, and Retrieval
Cappallo, S., Svetlichnaya, S., Garrigues, P. J., Mensink, T., & Snoek, C. G. M. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2862363 · Cited by 67 · see citing works
MeetingCoach: An Intelligent Dashboard for Supporting Effective & Inclusive Meetings
Samrose, S., McDuff, D., Sim, R., Suh, J., Rowan, K., Hernandez, J., Rintel, S., Moynihan, K., & Czerwinski, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445615 · Cited by 72 · see citing works
In-classroom learning analytics based on student behavior, topic and teaching characteristic mining
Yang, B., Yao, Z., Lu, H., Zhou, Y., & Xu, J. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2019.11.023 · Cited by 49 · see citing works
Towards More Comprehensive Listening Behavior: Beyond the Bobble Head
Wang, Z., Lee, J., & Marsella, S. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23974-8_24 · Cited by 35 · see citing works
Self-citationModeling Feedback in Interaction With Conversational Agents—A Review
Axelsson, A., Buschmeier, H., & Skantze, G. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.744574 · Cited by 33 · see citing works
“In the field of conversational agents, head gesture generation and evaluation, has, however, mainly focussed on head movements (including nods) as co-speech speaker—in contrast to listener— behavior (e.g., Lee and Marsella, 2010; Ding et al., 2013).”
BackgroundModeling multimodal behaviors from speech prosody,
Ding, Y., Pelachaud, C., & Artières, T. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40415-3_19 · Cited by 32 · see citing works
From Non-verbal Signals Sequence Mining to Bayesian Networks for Interpersonal Attitudes Expression
Chollet, M., Ochs, M., & Pélachaud, C. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-09767-1_15 · Cited by 32 · see citing works
The function of face gestures and head movements in spontaneous humorous communication
Navarro, M. B. (2019) · European Journal of Humour Research · doi:10.7592/ejhr2019.7.2.bujan · Cited by 26 · see citing works
Behavioral Analysis of Kinetic Telepresence for Small Symmetric Group-to-Group Meetings
Otsuka, K. (2017) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2017.2771396 · Cited by 21 · see citing works
Modeling Speaker Behavior: A Comparison of Two Approaches
Lee, J., & Marsella, S. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33197-8_17 · Cited by 20 · see citing works
Self-citationAn Expressive Virtual Audiencewith Flexible Behavioral Styles
Kang, N., Brinkman, W., van Riemsdijk, M. B., & Neerincx, M. A. (2013) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2013.2297104 · Cited by 20 · see citing works
Multi-party, multi-role comprehensive listening behavior
Wang, Z., Lee, J., & Marsella, S. (2013) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-012-9215-8 · Cited by 16 · see citing works
Self-citationTraining Socially Engaging Robots: Modeling Backchannel Behaviors with Batch Reinforcement Learning
Hussain, N., Erzin, E., Sezgin, T. M., & Yemez, Y. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3190233 · Cited by 16 · see citing works
A Multimodal Corpus Approach to the Design of Virtual Recruiters
Chollet, M., Ochs, M., Clavel, C., & Pélachaud, C. (2013) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2013.10 · Cited by 11 · see citing works
A Platform for Building Mobile Virtual Humans
Feng, A., Leuski, A., Marsella, S., Casas, D., Kang, S., & Shapiro, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-21996-7_33 · Cited by 8 · see citing works
Self-citationAudio-Visual Prediction of Head-Nod and Turn-Taking Events in Dyadic Interactions
Türker, B. B., Erzin, E., Yemez, Y., & Sezgin, T. M. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2215 · Cited by 8 · see citing works
Attitude Modeling for Virtual Character Based on Temporal Sequence Mining
Dermouche, S., & Pélachaud, C. (2018) · International Conference on Movement and Computing (MOCO) · doi:10.1145/3212721.3212806 · Cited by 7 · see citing works
The Role of Different Types of Conversations for Meeting Success
Zhou, K., Constantinides, M., Aiello, L. M., Joglekar, S. P., & Quercia, D. (2021) · IEEE Pervasive Computing · doi:10.1109/mprv.2021.3115879 · Cited by 7 · see citing works
Body Movements Generation for Virtual Characters and Social Robots
Beck, A., Yumak, Z., & Magnenat‐Thalmann, N. (2017) · Cambridge University Press eBooks · doi:10.1017/9781316676202.020 · Cited by 6 · see citing works
Online nod detection in human-robot interaction
Wall, E., Schillingmann, L., & Kümmert, F. (2017) · IEEE RO-MAN · doi:10.1109/roman.2017.8172396 · Cited by 6 · see citing works
Bayramoğlu, Ö. Z., Erzin, E., Sezgin, T. M., & Yemez, Y. (2021) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3462244.3479944 · Cited by 6 · see citing works
From embodied metaphors to metaphoric gestures.
Lhommet, M., & Marsella, S. (2016) · eScholarship (California Digital Library) · Cited by 5 · see citing works
Self-citationHuman Capacities for Emotion Recognition and their Implications for Computer Vision
Liebold, B., Richter, R., Teichmann, M., Hamker, F. H., & Ohler, P. (2015) · i-com · doi:10.1515/icom-2015-0032 · Cited by 5 · see citing works
Compensis, P. (2026) · Journal of Nonverbal Behavior · doi:10.1007/s10919-026-00500-y · Cited by 3 · see citing works
Emotionally Augmented Storytelling Agent
Lee, S., Johnson, A. E., Leigh, J., Luc Renambot, B., Jones, S., & Eugenio, B. D. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-21996-7_53 · Cited by 2 · see citing works
Le common ground et l’humour : comment le sens humoristique est créé et interprété en interaction
Tabacaru, S. (2025) · Cahiers de praxématique · doi:10.4000/147hw · Cited by 1 · see citing works
Computational model for interpersonal attitude expression
Dermouche, S. (2016) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2993148.2997619 · Cited by 1 · see citing works
Multimodal human-machine interaction including virtual humans or social robots
Magnenat‐Thalmann, N., Thalmann, D., & Yumak, Z. (2014) · SIGGRAPH Asia Courses · doi:10.1145/2659467.2675052
2018 Index IEEE Transactions on Multimedia Vol. 20
(2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2884393
Morency, A. S. A. S. M. V. H. L.-P. (2013) · CRC Press eBooks · doi:10.1201/b15477-17
Multimodal prediction of head nods in dyadic conversations
Türker, B. B., Sezgin, T. M., Yemez, Y., & Erzin, E. (2018) · Signal Processing and Communications Applications Conference (SIU) · doi:10.1109/siu.2018.8404737
Modeling Human Communication Dynamics for Virtual Human
Morency, L., Shapiro, A., & Marsella, S. (2013) · CRC Press eBooks · doi:10.1201/b15477-15
Self-citationDermouche, S. (2019) · theses.fr (ABES) · doi:10.70675/7287fc75z73edz4e7fzafdaz65bd3f23e3e8
Intent-Aware Backchanneling for Active Listening in Human-Robot Interaction
Sun, Y., Leusmann, J., & Hedderich, M. A. (2026) · Mensch und Computer (MuC) · doi:10.1145/3820253.3831400
- Authors
- Lee, J., & Marsella, S. C.
- Affiliations
- University of Southern California (US)
- ORCID
- Marsella, Stacy C.
- Venue
- IEEE Transactions on Multimedia
- Published
- 2010-10
- Volume
- 12
- Issue
- 6
- Pages
- 552-562
- ISSN
- 1520-9210 (ISSN-L)
- OpenAlex
- W2161956125
- Cited by
- 37
- References
- 43
- Citations by year
- 2026: 2 · 2025: 1 · 2022: 3 · 2021: 4 · 2019: 3 · 2018: 5 · 2017: 3 · 2016: 2 · 2015: 3 · 2014: 2 · 2013: 7 · 2012: 1
- FWCI
- 3.05
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Keywords
- head nodding, nonverbal communication, virtual agents, affective features, hidden Markov models, gesture prediction, feature selection, naturalness perception, face-to-face communication
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
