CoChat
Sign Up
Cited by 249Social Robot Interaction and HRISpeech and dialogue systemsAI in Service Interactions

Voice in Human–Agent Interaction: A Survey

Open access
Seaborn, Katie, Miyake, Norihisa P., Pennefather, Peter, Otake-Matsuura, Mihoko·2022·ACM Computing Surveys
DOI 10.1145/3386867
Open this paper in CoChat — free accountOpen PDF

Abstract

Social robots, conversational agents, voice assistants, and other embodied AI are increasingly a feature of everyday life. What connects these various types of intelligent agents is their ability to interact with people through voice. Voice is becoming an essential modality of embodiment, communication, and interaction between computer-based agents and end-users. This survey presents a meta-synthesis on agent voice in the design and experience of agents from a human-centered perspective: voice-based human–agent interaction (vHAI). Findings emphasize the social role of voice in HAI as well as circumscribe a relationship between agent voice and body, corresponding to human models of social psychology and cognition. Additionally, changes in perceptions of and reactions to agent voice over time reveals a generational shift coinciding with the commercial proliferation of mobile voice assistants. The main contributions of this work are a vHAI classification framework for voice across various agent forms, contexts, and user groups, a critical analysis grounded in key theories, and an identification of future directions for the oncoming wave of vocal machines.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3386867

Cites

176 of 176 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Research on CAs is a rapidly growing field, with an increase in publications over the last few decades, especially fueled through the development of CAs such as Amazon's Alexa that build upon voice as a modality.

    “Research on such agents is a rapidly growing field, with an increase in publications over the last few decades, especially fueled through the development of CAs such as Amazon's Alexa that build upon voice as a modality, so called voice bots (Schmitt et al., 2021;Seaborn et al., 2022;Kendall et al., 2020).”

    From Charting the Evolution and Future of Conversational Agents: A Research Agenda Along Five Waves and New Frontiers (2024) · cites this paper as Seaborn et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 17

  1. Lawrence W. Barsalou, Paula M. Niedenthal, Aron K. Barbey, and Jennifer A. Ruppert. 2003. Social embodiment. In The Psychology of Learning and Motivation: Advances in Research and Theory, Brian H. Ross (ed.). Academic Press, San Diego, CA, 43-92.
  2. Cynthia Breazeal, Kerstin Dautenhahn, and Takayuki Kanda. 2016. Social robotics. In Springer Handbook of Robotics. Springer, Cham, Switzerland, 1935-1972. https://doi.org/10.1007/978-3-319-32552-1_72 · DOI
  3. John W. Creswell and Cheryl N. Poth. 2016. Qualitative Inquiry and Research Design: Choosing Among Five Approaches (4th ed.). Sage, Thousand Oaks, CA.
  4. Nils Dahlbäck, Seema Swamy, Clifford Nass, Fredrik Arvidsson, and Jörgen Skågeby. 2001. Spoken interaction with computers in a native or non-native language-Same or different? In Proceedings of the International Conference on Human-Computer Interaction (INTERACT'01). 294-301.
  5. Andrea L. Guzman. 2016. Making AI safe for humans: A conversation with Siri. In Socialbots and Their Friends: Digital Media and the Automation of Sociality, R. W. Gehl and M. Bakardjieva (eds.). Routledge, New York, NY, 69-85.
  6. Charles F. Hockett and Charles D. Hockett. 1960. The origin of speech. Sci. Am. 203, 3 (1960), 88-97.
  7. IEEE Robotics and Automation Society. 2019. Robot ethics. IEEE Robot. Autom. Soc. Retrieved September 12, 2019 from https://www.ieee-ras.org/robot-ethics.
  8. Swati Johar. 2016. Emotion, Affect and Personality in Speech: The Bias of Language and Paralanguage. Springer Inter- national Publishing. Retrieved September 13, 2019 from https://www.springer.com/us/book/9783319280455.
  9. Jody Kreiman, Diana Vanlancker-Sidtis, and Bruce R. Gerratt. 2003. Defining and measuring voice quality. In Pro- ceedings of the Conference on Voice Quality: Functions, Analysis, and Synthesis. ISCA, 115-120. Retrieved December 17, 2019 from https://www.isca-speech.org/archive_open/voqual03/voq3_115.html.
  10. Jon A. Krosnick and Stanley Presser. 2010. Question and questionnaire design. In Handbook of Survey Research (2nd ed.). Emerald Group Publishing, Bingley, UK, 263-314.
  11. Martina Mara, Simon Schreibelmayr, and Franz Berger. 2020. Hearing a nose? User expectations of robot appear- ance induced by different robot voices. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI'20). ACM, 355-356. DOI:https://doi.org/10.1145/3371382.3378285 · DOI
  12. Robert R. McCrae and Paul T. Costa, Jr. 2008. A five-factor theory of personality. In Handbook of Personality: Theory and Research (3rd ed.), Oliver P. John, Richard W. Robins and Lawrence A. Pervin (eds.). The Guildford Press, New York/London, 159-181.
  13. Paul Milgram and Fumio Kishino. 1994. A taxonomy of mixed reality visual displays. IEICE Trans. Inf. Syst. E77-D, 12 (1994), 1321-1329.
  14. Kevin L. Nadal. 2017. The SAGE encyclopedia of psychology and gender. In Neurosexism. SAGE Publications, Thou- sand Oaks, CA, 1243-1246.
  15. Clifford Nass, Ulla Foehr, Scott Brave, and Michael Somoza. 2001. The effects of emotion of voice in synthesized and recorded speech. In Proceedings of the AAAI Symposium Emotional and Intelligent II. AAAI.
  16. Busra Sarigul, Imge Saltik, Batuhan Hokelek, and Burcu A. Urgen. 2020. Does the appearance of an agent affect how we perceive his/her voice? Audio-visual predictive processes in human-robot interaction. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRICompanion'20). ACM, 430-432. DOI:https: //doi.org/10.1145/3371382.3378302 · DOI
  17. Ari Schlesinger, W. Keith Edwards, and Rebecca E. Grinter. 2017. Intersectional HCI: Engaging identity through gender, race, and class. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, 5412- 5427. DOI:https://doi.org/10.1145/3025453.3025766 · DOI · Paper page

Cited by

Sorted by citations

Related

Authors
Seaborn, K., Miyake, N. P., Pennefather, P., & Otake-Matsuura, M.
Affiliations
  • Tokyo Institute of Technology (JP)
  • RIKEN Center for Advanced Intelligence Project (JP)
  • gDial Inc., Toronto, Ontario, Canada
  • RIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan
ORCID
Venue
ACM Computing Surveys
Publisher
Association for Computing Machinery (ACM)
Published
2021-05-03
Volume
54
Issue
4
Pages
1-43
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
DOI
10.1145/3386867
OpenAlex
W3159367349
Cited by
249
References
176
Citations by year
2026: 59 · 2025: 69 · 2024: 50 · 2023: 37 · 2022: 31 · 2021: 3
FWCI
29.64
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/publication-rights-and-licensing-policy (vor)
Locations
ACM Computing Surveys (publishedVersion)
Funders
Japan Society for the Promotion of Science (20H05022,20H05574)
Topics
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in Service Interactions
Keywords
social robots, chatbots, voice assistants, embodied AI, human-centered design, classification framework

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review