Towards building an attentive artificial listener: on the perception of attentiveness in audio-visual feedback tokens,
Open accessAbstract
Current dialogue systems typically lack a variation of audio-visual feedback tokens. Either they do not encompass feedback tokens at all, or only support a limited set of stereotypical functions. However, this does not mirror the subtleties of spontaneous conversations. If we want to be able to build an artificial listener, as a first step towards building an empathetic artificial agent, we also need to be able to synthesize more subtle audio-visual feedback tokens. In this study, we devised an array of monomodal and multimodal binary comparison perception tests and experiments to understand how different realisations of verbal and visual feedback tokens influence third-party perception of the degree of attentiveness. This allowed us to investigate i) which features (amplitude, frequency, duration...) of the visual feedback influences attentiveness perception; ii) whether visual or verbal backchannels are perceived to be more attentive iii) whether the fusion of unimodal tokens with low perceived attentiveness increases the degree of perceived attentiveness compared to unimodal tokens with high perceived attentiveness taken alone; iv) the automatic ranking of audio-visual feedback token in terms of conveyed degree of attentiveness.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Oertel et al. (2016) were able to identify features that led to an increase of the perceived level of attentiveness.
“Using this method they were able to identify features that led to an increase of the perceived level of attentiveness.”
From Modeling Feedback in Interaction With Conversational Agents—A Review (2022) · cites this paper as Oertel et al., 2016
For a head nod to communicate attentiveness, it should be rather long, have multiple oscillations and be overall more energetic.
“The result shows that for a head nod to communicate attentiveness, it should be rather long, have multiple oscillations and be overall more energetic.”
From Modeling Feedback in Interaction With Conversational Agents—A Review (2022) · cites this paper as Oertel et al., 2016
Cites
Praat: Doing Phonetics by Computer
(2011) · Ear and Hearing · doi:10.1097/aud.0b013e31821473f7 · Cited by 8,004 · see citing works
Praat : doing phonetics by computer (version 4.4.24)
Boersma, P. (2006) · Medical Entomology and Zoology · Cited by 1,109 · see citing works
Proceedings of Interspeech 2013
Lu, L., Ghoshal, A. K., & Renals, S. J. (2013) · Proceedings - International Symposium on Computer Architecture/Proceedings · Cited by 1,003 · see citing works
Embodiment in conversational interfaces
Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L. W., Chang, K., Vilhjálmsson, H. H., & Yan, H. (1999) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/302979.303150 · Cited by 569 · see citing works
Gesture and speech in interaction: An overview
Wagner, P., Malisz, Z., & Kopp, S. (2013) · Speech Communication · doi:10.1016/j.specom.2013.09.008 · Cited by 459 · see citing works
Gratch, J., Okhmatovskaia, A., Lamothe, F., Marsella, S., Morales, M., van der Werf, R. J., & Morency, L.-P. (2006) · Lecture Notes in Computer Science · doi:10.1007/11821830_2 · Cited by 247 · see citing works
The effect of head-nod recognition in human-robot conversation,
Sidner, C. L., Lee, C., Morency, L.-P., & Forlines, C. (2006) · Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction · doi:10.1145/1121241.1121291 · Cited by 107 · see citing works
Skantze, G., & Moubayed, S. A. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388698 · Cited by 94 · see citing works
Classifying subject ratings of emotional speech using acoustic features
Liscombe, J., Venditti, J. J., & Hirschberg, J. (2003) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.2003-306 · Cited by 94 · see citing works
Geometric Generative Gaze Estimation (G3E) for Remote RGB-D Cameras
Mora, K. A. F., & Odobez, J. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.229 · Cited by 94 · see citing works
Self-citationClassifying Subject Ratings of Emotional Speech Using Acoustic Features
Liscombe, J., Venditti, J. J., & Hirschberg, J. (2003) · doi:10.7916/d8f76mwv · Cited by 90 · see citing works
Bousmalis, K., Méhu, M., & Pantić, M. (2012) · Image and Vision Computing · doi:10.1016/j.imavis.2012.07.003 · Cited by 78 · see citing works
Exploring Turn-taking Cues in Multi-party Human-robot Discussions about Objects
Skantze, G., Johansson, M., & Beskow, J. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2820749 · Cited by 76 · see citing works
Multimodal Backchannels for Embodied Conversational Agents
Bevacqua, E., Pammi, S., Hyniewska, S., Schröder, M., & Pélachaud, C. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15892-6_21 · Cited by 56 · see citing works
Backchannels: Quantity, Type and Timing Matters
Poppe, R. W., Truong, K. P., & Heylen, D. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23974-8_25 · Cited by 41 · see citing works
Vilhjálmsson, H. H., Þórisson, K. R., Marsella, S., & Kopp, S. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23974-8 · Cited by 36 · see citing works
Towards More Comprehensive Listening Behavior: Beyond the Bobble Head
Wang, Z., Lee, J., & Marsella, S. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23974-8_24 · Cited by 35 · see citing works
Multi-party focus of attention recognition in meetings from head pose and multimodal contextual cues
Ba, S., & Odobez, J. (2008) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2008.4518086 · Cited by 34 · see citing works
Self-citationA multimodal analysis of vocal and visual backchannels in spontaneous dialogs,
Truong, K. P., Poppe, R., Kok, I. D., & Heylen, D. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-744 · Cited by 33 · see citing works
Oertel, C., Włodarczak, M., Edlund, J., Wagner, P., & Gustafson, J. (2012) · Interspeech (USB) · doi:10.21437/interspeech.2012-132 · Cited by 31 · see citing works
Self-citationOertel, C., Mora, K. A. F., Sheikhi, S., Odobez, J., & Gustafson, J. (2014) · doi:10.1145/2666242.2666251 · Cited by 24 · see citing works
Self-citationDeciphering the Silent Participant
Oertel, C., Mora, K. A. F., Gustafson, J., & Odobez, J. (2015) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2818346.2820759 · Cited by 24 · see citing works
Self-citationThe Nonverbal Context of Verbal Listener Responses
Rosenfeld, H. M., & HANCKS, M. (1980) · Mouton eBooks · doi:10.1515/9783110813098.193 · Cited by 24 · see citing works
The ALICO corpus: analysing the active listener
Malisz, Z., Włodarczak, M., Buschmeier, H., Skubisz, J., Kopp, S., & Wagner, P. (2016) · Language Resources and Evaluation · doi:10.1007/s10579-016-9355-6 · Cited by 23 · see citing works
Head Nod Detection from a Full 3D Model
Chen, Y., Yu, Y., & Odobez, J. (2015) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2015.75 · Cited by 22 · see citing works
Self-citationAnnotating meaning of listener vocalizations for speech synthesis
Pammi, S., & Schroder, M. (2009) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2009.5349568 · Cited by 20 · see citing works
Semi-supervised methods for exploring the acoustics of simple productive feedback
Neiberg, D., Salvi, G., & Gustafson, J. (2013) · Speech Communication · doi:10.1016/j.specom.2012.12.007 · Cited by 17 · see citing works
Self-citationProsodic Characteristics of Feedback Expressions in Distracted and Non-distracted Listeners
Malisz, Z., Włodarczak, M., Buschmeier, H., Kopp, S., & Wagner, P. (2012) · PUB – Publications at Bielefeld University (Bielefeld University) · doi:10.5281/zenodo.3234686 · Cited by 5 · see citing works
Flagged first, then by citations
2 of the 30 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A Bayesian Theory of Mind Approach to Nonverbal Communication
Lee, J. J., Sha, F., & Breazeal, C. (2019) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2019.8673023 · Cited by 39 · see citing works
Otsuka, K., & Tsumori, M. (2020) · IEEE Access · doi:10.1109/access.2020.3041672 · Cited by 38 · see citing works
HeadFusion: 360 Head Pose Tracking Combining 3D Morphable Model and 3D Reconstruction
Yu, Y., Mora, K. A. F., & Odobez, J. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2841403 · Cited by 35 · see citing works
Self-citationCho, E., Motalebi, N., Sundar, S. S., & Abdullah, S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555164 · Cited by 35 · see citing works
Modeling Feedback in Interaction With Conversational Agents—A Review
Axelsson, A., Buschmeier, H., & Skantze, G. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2022.744574 · Cited by 33 · see citing works
“Oertel et al. (2016) synthesized head nods from data and had them rated for perceived attentiveness of an agent.”
BackgroundRobust and Accurate 3D Head Pose Estimation through 3DMM and Online Head Model Reconstruction
Yu, Y., Mora, K. A. F., & Odobez, J. (2017) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2017.90 · Cited by 30 · see citing works
Self-citationTowards an Engagement-Aware Attentive Artificial Listener for Multi-Party Interactions
Oertel, C., Jonell, P., Kontogiorgos, D., Mora, K. A. F., Odobez, J., & Gustafson, J. (2021) · Frontiers in Robotics and AI · doi:10.3389/frobt.2021.555913 · Cited by 20 · see citing works
Self-citationCan Alexa be your Therapist?: How Back-Channeling Transforms Smart-Speakers to be Active Listeners
Motalebi, N., Cho, E., Sundar, S. S., & Abdullah, S. (2019) · Companion Publication of the 2019 Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/3311957.3359502 · Cited by 15 · see citing works
Multimodal User Feedback During Adaptive Robot-Human Presentations
Axelsson, A., & Skantze, G. (2022) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.741148 · Cited by 14 · see citing works
Mimebot—Investigating the Expressibility of Non-Verbal Communication Across Agent Embodiments
Alexanderson, S., O’Sullivan, C., Neff, M., & Beskow, J. (2017) · ACM Transactions on Applied Perception · doi:10.1145/3127590 · Cited by 10 · see citing works
Rasouli, S., Ghafurian, M., & Dautenhahn, K. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3718084 · Cited by 9 · see citing works
Crowd-Powered Design of Virtual Attentive Listeners
Jonell, P., Oertel, C., Kontogiorgos, D., Beskow, J., & Gustafson, J. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67401-8_21 · Cited by 5 · see citing works
Self-citationWhat Do People Think of Social Robots and Voice Agents as Public Speaking Coaches?
Forghani, D., Ghafurian, M., Rasouli, S., Nehaniv, C. L., & Dautenhahn, K. (2023) · IEEE RO-MAN · doi:10.1109/ro-man57019.2023.10309583 · Cited by 5 · see citing works
Huang, H., Kimura, S., Kuwabara, K., & Nishida, T. (2020) · Multimodal Technologies and Interaction · doi:10.3390/mti4020015 · Cited by 4 · see citing works
Crowdsourced Multimodal Corpora Collection Tool
Jonell, P., Oertel, C., Kontogiorgos, D., Beskow, J., & Gustafson, J. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/55tysysg2bbk · Cited by 4 · see citing works
Self-citationTarnec, H. L., Augereau, O., Bevacqua, E., & Loor, P. D. (2024) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/3656650.3656684 · Cited by 3 · see citing works
Evaluating people's perceptions of an agent as a public speaking coach
Forghani, D., Ghafurian, M., Rasouli, S., Nehaniv, C. L., & Dautenhahn, K. (2024) · Paladyn Journal of Behavioral Robotics · doi:10.1515/pjbr-2024-0004 · Cited by 3 · see citing works
Speaker Motion Patterns during Self-repairs in Natural Dialogue
Özkan, E. E., Gurion, T., Hough, J., Healey, P. G. T., & Jamone, L. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536220.3563684 · Cited by 1 · see citing works
Using crowd-sourcing for the design of listening agents: challenges and opportunities
Oertel, C., Jonell, P., Haddad, K. E., Székely, É., & Gustafson, J. (2017) · ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents (ISIAA) · doi:10.1145/3139491.3139499
Self-citationSystem Development and Evaluation of a Social Robot as a Public Speaking Coach
Forghani, D., Rasouli, S., Ghafurian, M., Jouaiti, M., Nehaniv, C. L., & Dautenhahn, K. (2025) · International Journal of Social Robotics · doi:10.1007/s12369-025-01320-8
- Authors
- Oertel, C., Lopes, J., Yu, Y., Mora, K. A. F., Gustafson, J., Black, A. W., & Odobez, J.-M.
- Affiliations
- KTH Royal Institute of Technology (SE)
- École Polytechnique Fédérale de Lausanne (CH)
- Idiap Research Institute (CH)
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 18th ACM International Conference on Multimodal Interaction
- Publisher
- ACM
- Published
- 2016-10-31
- Pages
- 21-28
- OpenAlex
- W2548091155
- Cited by
- 21
- References
- 30
- Citations by year
- 2025: 2 · 2024: 2 · 2023: 1 · 2022: 4 · 2021: 1 · 2020: 2 · 2019: 2 · 2018: 2 · 2017: 4 · 2016: 1
- FWCI
- 1.62
- Type
- conference paper
- Language
- en
- Open access
- green
- infoscience.epfl.ch
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- Funders
- European Commission
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
- Horizon 2020 Framework Programme
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Keywords
- backchannels, multisensory perception, third-party perceptions, dialogue systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
