The State of Speech in HCI: Trends, Themes and Challenges

“Inductive Thematic Analysis (Braun & Clarke, 2006) was conducted on the final search results to categorize the research themes discussed in each publication.”

“A key debate around the role of partner models seems to be emerging in HCI based publications, which reflects debates on language production more generally (Brennan & Clark, 1996; Horton & Keysar, 1996; Keysar, Barr, & Horton, 1998).”

“Approaches commonly used in interaction design research such as cultural probes (DeHaemer & Wallace, 1992), contextual inquiry (Holtzblatt & Jones, 1993), and participatory design more generally (Schuler & Namioka, 1993) tended not to be used in the research we reviewed.”

“This was to avoid studies where speech is confounded by issues of embodiment, such as gesture and emotive facial expressions (Breazeal, 2003), that can affect users’ interactions and experiences (Bruce, Nourbakhsh, & Simmons, 2002; Kuno et al., 2007).”

“Our review indicates that there has been some recent progress towards this (e.g. Luger & Sellen, 2016; Porcheron et al., 2017; Cowan et al., 2017).”

“In WoZ studies participants are led to believe they are interacting with a system, however, a (usually unseen) confederate controls the system’s output (Dahlbäck, Jönsson, & Ahrenberg, 1993).”

“This was also found by prior literature reviews of CAs [41, 139].”

“Unlike other forms of human-computer interfaces, there is little consensus as to best practice for the design of conversational agents [2].”

“This is consistent with further studies in which the participants question the relevance of a friendship-like relationship with machines, as well as a human-like CA design (Clark et al., 2019b; Doyle et al., 2019).”

“[9] note that evaluation of speech interfaces in real-world contexts is limited.”

“Speech, as a critical interface for human-computer interaction, can significantly enhance both interaction efficiency and user experience (Clark et al., 2019).”

“Note that this paper introduces our framework, but the task of filling it in is far from complete: the most obvious omission is around how linguistic content affects speech-based interaction, which is thoroughly studied in [16].”

“Validated metrics to quantify user perceptions are needed in the field [11].”

“They are conversational agents available through a number of devices such as smartphones, computers, and smart speakers [16, 68], and are widely used in a number of contexts such as domestic [68] and automotive settings [9].”

“But thus far, there has been little guidance of how to do this [12] and what the repercussions of VUI voices might be.”

“HCI is a rapidly growing field that presents both exciting new possibilities and formidable new obstacles (Clark et al., 2019; Ma et al., 2019; Vuletic et al., 2019).”

“[27] conducted a systematic review of 99 articles across a range of HCI publication venues, focusing on non-embodied CAs.”

“The proliferation of voice based Intelligent Personal Assistants (IPAs) like Google Assistant, on smart speakers and smartphones, has made speech a common interaction modality [8].”

“[14] offer a comprehensive survey on multimodal techniques incorporating speech input.”

“ACM ISBN 978-1-4503-8096-6/21/05. https://doi.org/10.1145/3411764.3445206 CCS CONCEPTS • Human-centered computing → Natural language interfaces; HCI theory, concepts and models; Interaction design theory, concepts and paradigms; • Applied computing → Psychology.”

“This is an adapted, participatory twist to Wizard of Oz approaches commonly used to showcase and demonstrate novel voice user interfaces [12].”

“The fragmented nature of speech-based CA work, in particular, has recently been addressed [29,78].”

“With the recent advances in speech recognition technology [1, 44, 87, 91, 108], today’s voice-based commercial products [23, 48, 62, 116–118, 120] can perform streaming, highaccuracy, low-latency speech recognition [15, 68] to revolutionize human-computer interaction [29].”

“WoZ could be an especially good starting point[2].”

“[4], [26] have enabled the deployment of voice-based services such as dictation, voice search and voice assistants in our daily life [15].”

“This recent and rapid progress has triggered a renewed interest in research on “the voice of the machine” [9,45].”

“Siri, Google Assistant, Amazon Echo) it is imperative that we develop an understanding of the mechanisms that drive language choices in HMD interactions [13].”

“Clark et al. [15] have summarized empirical research comparing speech versus graphical interfaces on user performance and experience, which has shown mixed results.”

“…from the majority of previous studies that investigated humanness in conversational agents relying on experiments or users ’ ex-post reports about their conversations with the agent (e.g., Clark et al., 2019; Doyle et al., 2019, 2021), we analyze real-world interactions between users and a chatbot.”

“With the rise of speech recognition and natural language processing (NLP) technologies, speech has become a prevalent modality to interact with digital devices [19].”

“speech, which is widely embedded in consumer devices and digital systems [22].”

“Suggest-ing a lack of sufcient guidelines, practitioners adopt graphical user interfaces (GUIs) best practices to account for the unique af-fordances that CUIs present [16, 46, 56].”

“In addition, speech-centric modeling has created numerous research topics in human behavior understanding, human-computer interface (HCI) [41], and social media analysis, involving several widely used speech modeling techniques like automatic speech recognition [110], speech emotion recognition [5], automatic speaker…”

“…items included were is reliable, gives security in information search, and respects people’s privacy, thereby incorporating common user (and designer) concerns regarding VBAs (Clark et al., 2019; Easwara Moorthy & Vu, 2015; Kinsella & Mutchler, 2020; Lei et al., 2018; Olson & Kemery, 2019).”

“Scholarly work about people’s experiences with voice assistants is often based on qualitative methods and comes from the field of human–computer interaction (see Clark et al., 2019, for a review).”

“This workshop aims to address current issues and potential benefits by bringing together conversational UX designers, medical informatics, HCI, machine learning, and design researchers investigating:”

“Furthermore, it is valuable to perform in-the-wild experimentation including focus groups on developed SLT applications, which can uncover issues and harms affecting user populations [51].”

“As voice interactions have become more commonplace, a growing body of work [19] has helped to illustrate when, where, why, how, and for whom voice can be most useful, and the key challenges and limitations of voice assistants.”

“As extensively noted in VUI literature, VUIs are not “conversational” in nature [24, 25, 31, 36, 45, 83].”

“For the current review, the list of terms was derived directly from similar papers that performed systematic reviews on speech-based and conversational interfaces [7, 16, 52].”

“, [17, 25]), which may not refect what actually happened during the interaction, or on experiments, in which people are given instructions on how to converse with the agent and are not pursuing their own situated goals [3, 79, 95]: nonetheless, if interaction is not driven by a real goal that refects the fulflment of…”

“Researchers have recently called for more work in voice design theory, particularly in specific contexts [10], and the above discussion can inform the extension of existing theory to users as producers of a voice.”

“face interaction Current work on language productionin speech interface interactionalmostuniversally observes thelanguage choices ofL1 speakers. Even then, the volume of work on this topic is limited [9], with a focus on comparing language production in interactions between human-machine and human-human interlocutors.…”

“Although automatic speech recognition (ASR) and text-to-speech (TTS) systems can be easily employed, the linguistic discrepancy between speech and text causes dialog inefficiencies and result in sub-optimal user experience [10, 11].”

“The WoZ method has been widely adopted in speech-based HCI studies [7, 12, 33].”

“Intelligent voice assistants, or IVAs, which are integrated into smart speakers such as Amazon’s Alexa, are rapidly gaining popularity and becoming an integral part of everyday life [1,2].”

“Although early work found that there was no interaction between listener gender and speech fdelity [39], it is important to determine if older speech research applies to newer speech technologies [12, 56].”

“3406139 [5], and attempts to move towards more conversational interactions with users [6], point to speech becoming more common in”

“Corresponding to a rise in voice assistant popularity, there has been a surge of interest within the HCI community in studying voice agents and other speech technology [15].”

“And yet, there exists a general consensus among both HCI and healthcare researchers that limited design-oriented work has been conducted concerning CAs to date [12, 25, 28].”

“Research on designing usable VUIs is more limited than for other interaction types, as only recently have significant inroads been made in the VUI consumer space [9].”

“For those who decides to interact with VUI, an important issue that affects user experience during the interaction is the conversation errors [7, 14], which require extra efforts for conversation recovery or lead to breakdowns [10].”

“A variety of vocal qualities have been explored in computer voice over the past several decades [10, 43, 46].”

“Combining subjective, behavioral, and physiological measures of trust not only enrich the mixed methods approach proposed here for studying trust, but they may also, in combination, allow for creation of robust validated measures which are lacking in conversational HCI in general [2].”

“Conversely, the phenomenon of commercial VAs is nascent and lacks design guidelines and research on metaphors [16].”

“This would also answer calls in the field for more theory-based understanding of speech interface interactions more generally [13].”

“As a result, the use of speech to interact with automated systems has grown in popularity (Clark et al., 2019; Lopatovska et al., 2020).”

“However, although subjective measures are widely deployed, questions have been raised about the reliability and validity of the largely bespoke questionnaires normally used in these studies [19, 24, 52, 80].”

“We derived the key-words directly from similar papers which performed meta-analyses on speech-based and conversational interfaces –the most relevant being a best paper from CHI 2019 which performed a large-scale survey on speech and voice interface papers in HCI [23] (see Table 1).”

“Consequently, how to measure the UX of the CAs is an urgent yet unsolved question that has been pointed out by several recent research too (Clark et al., 2019; Dutsinma et al., 2022; Guerino & Valentim, 2020; Silva & Canedo, 2022; Tubin et al., 2022).”

“While speakingwith and listening to computers has a long history within the field of human-computer interaction (HCI), these recent industrial and commercial trends have reinvigorated interest in the voice of the machine [28, 95].”

“Besides voice-based interaction [12] which is known to be limited to quiet environments and poses privacy issues [46, 95], research proposed leveraging the movements of other body parts.”

“…interaction field with its varied ways to conduct research with new technologies – hosting co-design workshops, facilitating interviews or focus groups, gathering feedback through prototype testing, implementing WOz experiments, and assessing established commercial systems (Clark et al., 2019).”

“Human-Computer Interaction (HCI) researchers have also explored the role that speech has on listeners’ perceptions of spoken interfaces’ credibility [11, 9, 8, 36], and some of them have done so concerning virtual humans [25, 33, 10].”

“Automatic speech recognition (ASR) [4] has received considerable attention and recently made significant progress with its applications in mobile computing [5], human– computer interaction [6,7], information retrieval [8], and assisted communication [9].”

“However, research highlights that voice interfaces require better theories and more design guidelines, due to persistent usability issues [14, 48].”

“Across the globe, the number of users relying upon modern-day speech applications, such as automatic speech recognition (ASR), speaker recognition and verification , speech synthesis , speech emotion recognition (SER), will keep on increasing [14].”

“Prior research on speech as a modality in human-computer interaction (HCI) has demonstrated its benefts in improving access to interactive systems [12].”

“With rapid commercialization, we now see a growing use of CUIs like Alexa in everyday places; research interests accompany this growing use [22, 55].”

“Second, this workshop is intended to further grow the Conversational User Interfaces (CUIs) community and scaffold this area at CHI.”

“Although the technical aspects of VUIs have been largely investigated, researchers agree on the stance that the user side of speech interaction is relatively less explored [8, 22, 47, 49].”

“Others have even questioned whether some conversational user interfaces are (or indeed should be) in effect conversational [7, 10– 12, 15, 25, 26, 29].”

“31 Furthermore, there is a lack of appropriate attention to the context of use, as UX evaluation questionnaires usually focus on perceived product qualities or emotions.”

“Voice assistants (VAs) have permeated our daily lives, being ubiquitous across a range of devices such as smartphones, smart speakers, and computers [10, 45], and in a variety of environments such as the”

“Despite this, the majority of research on CUIs seems to focus elsewhere, with only 15% of papers referenced in the recent review by Clark et al. [19] focusing on mobile interaction even after excluding embodied interaction with devices such as stand-alone smart speakers and robots.”

“Hence, from the user’s perspective since it helps in reducing their information redundancy or overload and create personalized offerings for them, thereby it makes the VAs suitable as life companions on which the users can trust and depend upon [52], [53], [54].”

“As voice-based interactive technologies increasingly become part of everyday life, weaknesses in speech-to-text systems are rapidly becoming a key bottleneck (Clark et al., 2019).”

“For example, studies of the potential of CUIs for accessibility in a survey by Clark et al. 2019 tend to focus on the potential of speech technologies to enable people with a range of impairments to complete a specific set of performance-measurable tasks (see e.g., [5, 61, 71, 72]).”

“In fact, recent reviews on voice assistants indicate a lack of concrete design guidance for the next generation of IPAs [12].”

“…works previously described analyzed the accuracy of recognition of spoken vocabulary and the performance of users when performing tasks with an IVA, also known as task performance (Clark et al., 2019), in order to assess how prepared the system is to understand the diversity of users ’ discourses.”

“Previous CUI research spans how to best design CUI interactions [4, 5, 9, 16, 17, 34, 44], such as tools and heuristics on voice-based interfaces [8, 36] and sociolinguistic considerations [9, 11, 27], as well as diverse ways to understand people’s difer-ent preferences or perceptions regarding…”

“…settings tends to refer to either canned speech (i.e., unchangeable pre-recorded speech samples) or synthesised speech, both seen in voice research (Nass and Brave, 2005; Kaur and Singh, 2023; Clark et al., 2019; Cambre and Kulkarni, 2019; Weinschenk and Barker, 2000; Kang and Heide, 1992).”

“Qualitative research, however, is rarely characterized as posi-tivist.”