The State of Speech in HCI: Trends, Themes and Challenges

“Inductive Thematic Analysis (Braun & Clarke, 2006) was conducted on the final search results to categorize the research themes discussed in each publication.”

“A key debate around the role of partner models seems to be emerging in HCI based publications, which reflects debates on language production more generally (Brennan & Clark, 1996; Horton & Keysar, 1996; Keysar, Barr, & Horton, 1998).”

“Approaches commonly used in interaction design research such as cultural probes (DeHaemer & Wallace, 1992), contextual inquiry (Holtzblatt & Jones, 1993), and participatory design more generally (Schuler & Namioka, 1993) tended not to be used in the research we reviewed.”

“This was to avoid studies where speech is confounded by issues of embodiment, such as gesture and emotive facial expressions (Breazeal, 2003), that can affect users’ interactions and experiences (Bruce, Nourbakhsh, & Simmons, 2002; Kuno et al., 2007).”

“Our review indicates that there has been some recent progress towards this (e.g. Luger & Sellen, 2016; Porcheron et al., 2017; Cowan et al., 2017).”

“In WoZ studies participants are led to believe they are interacting with a system, however, a (usually unseen) confederate controls the system’s output (Dahlbäck, Jönsson, & Ahrenberg, 1993).”

“Interactions with CUIs do not entirely mimic human interaction, leading some to wonder whether the notion of “conversation” itself in this domain needs to be reframed (Clark et al., 2019b; Porcheron et al., 2018; Reeves, 2019; Stokoe et al., 2020).”

“Clark et al. [56] analyzed that speech human–computer interaction work focuses on nine key topics: system speech production, modality comparison, user speech production, assistive technology and accessibility, design insight, and experiences with interactive voice response (IVR) systems.”

“As with its wider use through intelligent personal assistants in mobile phones and smart speaker devices [9, 21, 34], speech in VEs is also perceived as effective in facilitating multitasking, freeing up other modalities of control [35].”

“As Clark et al. [22] note, we can use a range of low and high fi delity approaches, including Wizard of Oz [87], where a human pretends to be the VA (iii) Embrace Deviancy .”

“[8] point out “A high proportion of research in our review used self-report ques-”

“Consequently, prior research focuses on the advancement of home automation in the context of intelligent assistance, including work on multimodality (cf. [42]), conversational assistance (cf. [12, 23], [45]), and the role of proac-tivity [56].”

“While technologies with voice-based interaction are often praised for benefts [10] in the home [52], in group interactions in public spaces [54], as well during activities such as driving [31], their use by cyclists seems to be somewhat muted.”

“How then might we evaluate the advice provided by such CAs? While there are agreed-upon techniques for the evaluation of conversational systems [4, 42], these have been developed to evaluate task-based system performance [13], using a mix of subjective and objective metrics.”

“However, this area has not yet been fully realized as a prominent means of human-computer interaction [2-4].”

“…• Identify key design challenges in the development of CUI devices, and particularly, the tools and practices which are key to the space of developing CUIs which are not yet in existence and require more research This workshop will build upon signifcant momentum in this area [5, 7, 9, 20–23 , 28 ].”

“While various studies investigated how some elements of conversation architecture affect aspects of users’ perceptions or discussed the speech specifics of agents as part of a broader analysis of the state of research [11, 23], there is no comprehensive analysis of conversation architecture elements that are relevant…”

“speech [5] or brain [29], hand has been widely utilized mainly due to its intuitiveness.”

“Although there are many robust scales, research shows that there are also many that lack reliability or validity testing [23].”

“A significant amount of CUI research has explored single-party interactions [3], often noting significant differences in how machine and human dialogue partners are perceived [6], how this impacts on language production in human-machine dyads [5], and how the design of existing CUI systems such as…”

“A recent review on speech interfaces reviewing 99 papers (Clark et al. 2019) concluded, amongst others, that most papers focused on usability, and not persuasive outcomes, and are rather diverse, or even fragmented.”

“UX research in the language services industry is not limited to interaction with MT systems, as we can see a proliferation of virtual assistants, smartphones and intelligent devices with which we communicate and interact on a daily basis (Clark et al., 2019).”

“This lack of consensus and continuity in how concepts are defined also echoes a general problem in human-machine dialogue (HMD) research, which has been highlighted in a number of recent reviews of research in the field [4, 5, 28, 29].”

“Yet, post-wimp (windows, icons, mouse, pointer) interfaces like VUI [8, 24, 35] or interfaces for Mixed Realities [18] still lack a uniform and acknowledged set of such design guides, in comparison to traditional GUI-based systems.”

“After that, we analyzed the self-reported qualitative data, especially focusing on difficulties, and we also analyzed quantitative data from users by using the IVR usability questionnaire, Speech User Interface Service Quality (SUISQ-MR) [9, 8, 4].”

“A consistent metaphorical framing provides a stable anchor for user expectations, supports long-term relationship building, and reduces the risk of disorienting users through unexpected behavioral shifts [18, 33, 34].”

“1 Introduction Due to the growth in popularity of Intelligent Personal Assistants (IPAs) such as Google Assistant and Amazon’s Alexa, using speech as a means of interface interaction is now well established [12].”

“Much effort has gone into identifying and realizing ideal forms of engagement and user experience (UX) with these technologies [17, 63], whilst also understanding people’s perceptions of VUIs as dialogue partners [21, 24], termed partner models [13, 14, 21, 24].”

“The use of speech interfaces is growing in popularity [18].”

“For this, we utilized the Wizard-of-Oz method, a well-established approach in HCI for exploring future technology [ 24, 41, 55, 88].”

“Clark, Leigh et al. [21] surveyed 68 research papers to identify the trends, themes, findings and methods of empirical research on speech interfaces in HCI.”

“Across speech research in HCI, some theoretical concepts and methods have started to be borrowed from other disciplines to explain user behaviour and inform speech interface design [5].”

“This field aims to understand and improve the fit between people and computers in situations where the interaction is based on speech [8, 53].”

“…that feature Conversational User Interfaces; CUIs) are gaining favour in addressing these issues by keeping the driver appraised of the driving situation, as this form of user-interaction is flexible, immediate, and rich in information (Clark et al., 2019c; Large et al., 2017; Velaga et al., 2021).”

“This paper’s core contribution is an understanding of how well mass media messaging is supported by academic design literature – that is, human-computer interaction (HCI) literature, which herein we refer to interchangeably with the terms ‘academic research,’ ‘academic findings,’ ‘academic evidence,’ ‘academic literat…”

“A recent review of the HCI literature concerning speech interfaces suggests a lack of empirical work in voice-based conversational interaction due to the barriers to evaluation in real-world contexts, thereby impairing our understanding of CA user experience (31).”

“Prior research on accessibility and audio-based interfaces emphasizes how disruption is detrimental to non-visual information access [24, 39, 59, 75, 99, 115].”

“Systematic literature reviews have been conducted on a wide variety of topics pertaining to chatbots, ranging from their design features (Ter Stal et al. 2020) to their speech (Clark et al. 2019).”

“centered studies to design more intuitive speech-based interfaces and (b) summarizing theoretical understandings of speech as a way to inform interface design [12].”

“Surveying the state of speech in HCI, Clark et al. identifes the need to devise theories of speech interface interaction [12].”

“Methodological and evaluation challenges have been noticed by several authors in VUI [13, 30, 42, 46, 52, 53], ARUI/VRUI [4, 6, 20,”

“For instance, Apple’s Siri 1 , Amazon’s Alexa 2 , and Google Assistant 3 have revolutionized how users interact with their devices, allowing for hands-free operation of smart-phones, smart home devices, and even vehicles [1], [2].”

“…eye gaze interaction, voice interaction, non-verbal vocal input, and brain-computer interfaces, Jankowski et al. state that novel input modalities will not replace the mouse, keyboards, and touchscreens as the major input modality for interacting with standard monitor devices [9, 17, 37, 46].”

“…(IPAs) – also referred to as virtual, digital, or, when it is the dominant interaction modality, voice assistants 1 – became practically omnipresent in the day-to-day lives of users through integration in personal computers, smart-phones, smartwatches, smart TVs, and smart speakers [5, 9, 20].”

“Consequently, VAs need reliable usability to prevent users from negative experience [4, 31, 32], and furthermore research to investigate the positive aspects of user experience, which might contribute to an enchanting, playful, meaningful, and engaging interaction [33].”

“Accessibility (mentioned in a recent review by Clark et al. (2019)) and broadcasting are additional areas.”

“The technological advancements and rapid growth of commercially available chatbots [6] and speech interfaces [5] - accessible through websites, mobile devices, in-car communication systems and smart speakers amongst others - makes the need for developing established theoretical and methodological frameworks in CUI int…”

“Speech-based interaction is a burgeoning area for research as a recent review by Clark et al. [9] reminded.”

“The centrality of speech is increasingly reflected in modern computing, from in-vehicle voice assistants and game-playing agents to voice-based robot control and computer use (Arora et al., 2025; Clark et al., 2019).”

“As a representative, voice assistants, a common type of CA, are now ubiquitous and available across a range of devices such as smartphones, computers, and smart speakers [18], utilized in diverse contexts, including smart home and automotive settings [1, 9, 54].”

“Users also interpret the voice phenomena they hear in ways beyond mere content, notably reading in cues to humanlikeness and artificiality [14, 44, 54].”

“For the speech interaction modality specifically, Clark et al. [12] conducted a thematic analysis to summarize nine primary research topics for HCI works with a focus in speech interfaces and identify gaps and challenges in speech interface research.”

“As such, alongside the move towards voice- and speech-based interaction [15, 34], gender bias in NLP is”

“However, understanding of what governs user language choices in such humanmachine dialogues (HMD) is sparse [14].”

“Virtual assistants, or intelligent personal assistants (IPAs), as they are frequently titled in literature [5], became omnipresent in recent years thanks to the now-common practice of integrating such services in personal computers, smartphones, smartwatches, smart TVs, and smart speakers [6, 18, 31].”

“However, further exploration is needed to properly recognize users’ intent in free-form conversation and implement suitable processing techniques such as context-aware topic modeling to enable an intuitive and effective communication environment [28].”

“For a full review of the “State of Speech in HCI”, we refer to Clark et al. [8].”

“Woz has been used to prototype applications of new technologies, such as VR apps and digital assistant apps (Jonell et al. 2017; Clark et al. 2019).”

“Voice interaction, a subject extensively studied across various domains including telephony, mobile applications, and in-vehicle systems [28, 68, 127] is fundamentally shaped by its inherent characteristics beyond mere linguistic content.”

“This reinforces the fear of social stigma, where non-native English-speaking academics feel pressure to conform to the linguistic expectations of HCI academic settings [31, 32], even when they might beneft from technological support.”

“Currently, machine learning tools have rapidly changed the voice synthesis landscape, making it possible for more people to play with, create and modify synthetic voices in new ways, exploring the breath of vocal expressions [4].”

“in research on how the use of voice controls affects the emotional evaluation of mobile phones as well as general underrepresentation of voice-controlled systems in hci research [2, 7].”

“The rapid growth of speech as an interaction modality has been driven by the uptake of smart speakers and smartphone-based assistants by consumers, thereby stimulating an increase in HCI research aiming to understand users’ preferences, challenges, and strategies related to their IPA use [1].”

“[2] did a systematic review on the status of speech interfaces in HCI.”

“In XR, early studies such as Put That There [5] have examined how speech and other interaction modalities (e.g., gesture) together clarify user intent.”

“Although trust in technology – including automotive systems – has been extensively studied, trust in in-vehicle VAs remains a relatively underexplored domain (Clark et al., 2019).”

“Speech interfaces allow users to issue commands verbally or dictate text input [12, 31].”

“Additionally, by considering speech for interaction, we can potentially make Human-Computer Interaction more interesting, realistic, and easier from the user’s perspective [9, 12] in several contexts.”

“Similar to SDSs, spoken language understanding (SLU), involves automatic speech recognition, natural language processing, understanding, and synthesis [6, 25].”

“Voice-based virtual assistants (VAs), in particular, can provide hands-free guidance, which is suitable for mindfulness meditation [8, 27].”

“facilitate comfort and rapport building in situations where forming a bond and building trust are important, such as in healthcare, education, or social programs [18], and can act as a strong initial enticement for people to adopt speech interfaces in the first place [37].”

“Even though many interactions with conversational technology were shown to be discouraging and disappointing [45], and the development of theories and work on speech interaction design proves to be challenging [14], relevancy does not decline.”

“Earlier researchers [13, 19] also emphasized the importance of re-evaluating earlier research in light of modern speech synthesis technology as it continues to improve.”

“We introduce the term of an ‘unremarked listener’ (Figure 3) role to refer to the previously used term ‘eavesdropper’ (Bell 1984).”

“Text and voice modalities offer different UX and may be used together (Clark et al., 2019; Seaborn et al., 2021).”

“The paper advocates for increased design efforts and extensive research on multi-user interaction settings (Klumpp et al., 2019; Clark et al., 2019).”

“With the advancement of speech technologies, the scope and expectations of speech-based interactions evolve into conversational interactions, over the last two decades (Clark et al., 2019; Cowan, 2017; Glass, 1999; Leigh Clark, 2019; Yankelovich, 1995).”

“No official conversation design principles or thorough VA studies have been done to inform conversation designers on language forms and discursive structure of speech output (Clark et al., 2019).”

“With regard to perceived understandability, users consider news with a slow speed to be more understandable and have a better user experience [8, 9].”

“Furthermore, the current voice interaction technology is not seamless but more like a task-based dialogue paradigm [3].”

“matter, to human organization or human activity as a whole [9, 10].”

“…uploaded to the cloud for spoken language understanding (SLU) is steadily increasing [1, 13, 2], particularly in ubiquitous wimpy devices where textual input is inconvenient [50, 19, 3], e.g., home automation devices [40], smartwatches [46], telehealth sensors [26] and smart factory sensors [35] .”

“Knowing the words being spoken (e.g., keyword spotting [53]) and the direction of the voice can be important for applications in improving situation awareness [79], AR/VR/MR [84], human-computer interaction [17, 55], and more.”

“NIME’24, 4–6 September, Utrecht, The Netherlands. in human-computer interaction [9, 5, 19] and at NIME [12].”