“Try, Try, Try Again:” Sequence Analysis of User Interaction Data with a Voice User Interface
The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
Voice User Interfaces (VUIs) pose challenges for users to learn the system’s supported features and commands. Users often rely on trial-and-error to navigate VUI dialogues and complete desired tasks. The order in which users try different commands contains vital information about how they learn. In this paper, we explore using sequence analysis techniques to reveal the patterns of tactics our participants (n = 50) used when interacting with an unfamiliar multi-modal VUI. We present the results of our sequence analysis, the top sequences used, and a cluster analysis of our participants on their usage of the top sequences. Our results indicate participants initially struggled with understanding the acceptable utterance structure and entities more so than utterance keywords. Additionally, we discuss our participants’ behavior differences and discuss the usefulness of this methodology of sequence analysis for HCI VUI research.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Specifically, in the personal assistance domain, individual studies focused on utilising either computer speakers [53, 54, 57, 67, 79], mobile devices [50] or smart speakers [26, 40].”
From System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping Review (2024) · cites this paper as [54]
In the citing paper
“For example, a study on the sequence patterns of users interacting with VUIs [54] found that users often add more details to their utterances to clarify their intent when they encounter a breakdown with a new task.”
From System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping Review (2024) · cites this paper as [54]
Citations
Cites
Agrawal, R., & Srikant, R. (2002) · Proceedings of the Eleventh International Conference on Data Engineering · doi:10.1109/icde.1995.380415 · Cited by 5,155 · cited 1×
“, trial-and-error), that they had sufficient variance over the population, and that they were minimally correlated with each other (so that each sub-sequence did not show a behavior already captured in another sub-sequence) [2, 41, 46]).”
BackgroundApplied Nonparametric Statistics
Wozniak, P. J., & Daniel, W. W. (1991) · Technometrics · doi:10.2307/1268795 · Cited by 4,987
Entropy-based algorithms for best basis selection
Coifman, R. R., & Wickerhauser, M. V. (1992) · IEEE Transactions on Information Theory · doi:10.1109/18.119732 · Cited by 3,188 · cited 1×
“The next step was to identify the most information-rich sub-sequences; for this, we applied Shannon Entropy [11] and an entropy score was calculated for each sequence.”
MethodLuger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176 · cited 2×
“HCI VUI research more commonly reviews and discusses the aggregated behavioral data of how users interact with a VUI [13, 31, 36] or observations of VUI interaction [10, 17, 33].”
Background"What can i help you with?": infrequent users' experiences of intelligent personal assistants
Cowan, B. R., Pantidi, N., Coyle, D., Morrissey, K., Clarke, P., Al-Shehri, S., Earley, D., & Bandeira, N. (2017) · Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services · doi:10.1145/3098279.3098539 · Cited by 351 · cited 1×
“For example, studies interview users to reflect on their VUI experience and behavior [4, 14, 33], conduct in-situ interviews to elicit user utterances [9, 10, 32], or label”
BackgroundUnderstanding the Long-Term Use of Smart Speaker Assistants
Bentley, F., Luvogt, C., Silverman, M., Wirasinghe, R., White, B., & Lottridge, D. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3264901 · Cited by 316 · cited 2×
“a collected corpus of VUI dialogues to analyze [4, 5, 19, 35, 38].”
BackgroundPatterns of entry and correction in large vocabulary continuous speech recognition systems
Karat, C.-M., Halverson, C., Horn, D., & Karat, J. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303160 · Cited by 275 · cited 1×
“Researchers have identified that VUIs’ low learnability imposes the second most frequent obstacles for users [35], immediately following Natural Language Processing (NLP) errors [18, 22, 27, 36].”
BackgroundInfluentialPatterns for How Users Overcome Obstacles in Voice User Interfaces
Myers, C., Furqan, A., Nebolsky, J., Caro, K., & Zhu, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173580 · Cited by 213 · cited 10×
“Additionally, in [35], we employ Markov Chain modeling to see the probability of one tactic following another.”
BackgroundMethodResultInfluentialSelf-citationGuns, swords and data: Clustering of player behavior in computer games in the wild
Drachen, A., Sifa, R., Bauckhage, C., & Thurau, C. (2012) · 2012 IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2012.6374152 · Cited by 180 · cited 1×
“Snapshot profiling is based on aggregate data but only provides information about the state of the users across the period of time covered by the snapshot [16, 40].”
BackgroundCorbett, E., & Weber, A. (2016) · Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services · doi:10.1145/2935334.2935386 · Cited by 148 · cited 3×
“These studies test various ways to improve the usability of VUIs through prompt and menu designs for discoverability of commands [13, 17, 31, 42], adaptive user models to adjust the system to users’ predicted needs [23, 43], recovery and repair strategies [30]; to name a few.”
Background
Research works printed but not linked: 12
- John Brooke . 1996. SUS A quick and dirty usability scale. Usability evaluation in industry 189, 194 ( 1996 ), 4–7. John Brooke. 1996. SUS A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7.
- William Jay Conover and Ronald L Iman . 1979 . On multiple-comparisons procedures. Los Alamos Sci. Lab. Tech. Rep. LA-7677-MS 1 (1979), 14 . William Jay Conover and Ronald L Iman. 1979. On multiple-comparisons procedures. Los Alamos Sci. Lab. Tech. Rep. LA-7677-MS 1 (1979), 14.
- 10.1145/3332167.3357103 · DOI
- Kristiina Jokinen , Kari Kanto , Antti Kerminen , and Jyrki Rissanen . 2004 . Evaluation of Adaptivity and User Expertise in a Speech-Based E-Mail System . Proc. of the 20th ACL International Conference on Computational LinguisticMarch 2016(2004) , 44–52. Kristiina Jokinen, Kari Kanto, Antti Kerminen, and Jyrki Rissanen. 2004. Evaluation of Adaptivity and User Expertise in a Speech-Based E-Mail System. Proc. of the 20th ACL International Conference on Computational LinguisticMarch 2016(2004), 44–52.
- Pavan Kantharaju , Katelyn Alderfer , Jichen Zhu , Bruce Char , Brian Smith , and Santiago Ontanón . 2018 . Tracing player knowledge in a parallel programming educational game . In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 14 . Pavan Kantharaju, Katelyn Alderfer, Jichen Zhu, Bruce Char, Brian Smith, and Santiago Ontanón. 2018. Tracing player knowledge in a parallel programming educational game. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 14.
- Sasha Makarovych , Alessandro Canossa , Julian Togelius , and Anders Drachen . 2018 . Like a DNA string: Sequence-based player profiling in Tom Clancy’s The Division. In Artificial Intelligence and Interactive Digital Entertainment Conference. York. Sasha Makarovych, Alessandro Canossa, Julian Togelius, and Anders Drachen. 2018. Like a DNA string: Sequence-based player profiling in Tom Clancy’s The Division. In Artificial Intelligence and Interactive Digital Entertainment Conference. York.
- Martin Porcheron , Joel E. Fischer , Stuart Reeves , and Sarah Sharples . 2018. Voice Interfaces in Everyday Life. CHI ’18 ( 2018 ), 1–12. https://doi.org/doi.org/10.1145/3173574.3174214 Martin Porcheron, Joel E. Fischer, Stuart Reeves, and Sarah Sharples. 2018. Voice Interfaces in Everyday Life. CHI ’18 (2018), 1–12. https://doi.org/doi.org/10.1145/3173574.3174214 · DOI · Paper page
- Rafet Sifa , Anders Drachen , and Christian Bauckhage . 2018. Profiling in games: Understanding behavior from telemetry. Social interactions in virtual worlds: An interdisciplinary perspective ( 2018 ). Rafet Sifa, Anders Drachen, and Christian Bauckhage. 2018. Profiling in games: Understanding behavior from telemetry. Social interactions in virtual worlds: An interdisciplinary perspective (2018).
- Stefan Ultes , Robert ElChab , and Wolfgang Minker . 2014. Application and evaluation of a conditioned hidden markov model for estimating interaction quality of spoken dialogue systems . In Natural Interaction with Robots, Knowbots and Smartphones . Springer , 303–312. Stefan Ultes, Robert ElChab, and Wolfgang Minker. 2014. Application and evaluation of a conditioned hidden markov model for estimating interaction quality of spoken dialogue systems. In Natural Interaction with Robots, Knowbots and Smartphones. Springer, 303–312.
- Josep Valls-Vargas , Santiago Ontanón , and Jichen Zhu . 2015 . Exploring player trace segmentation for dynamic play style prediction . In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 11 . Josep Valls-Vargas, Santiago Ontanón, and Jichen Zhu. 2015. Exploring player trace segmentation for dynamic play style prediction. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 11.
- G Wallner . 2002 . Sequential Pattern Mining Using Bitmaps . In Proceedings of the Eigth ACM SIGKDD International Conference of Knowledge Discovery and Data Mining. 429–435 . G Wallner. 2002. Sequential Pattern Mining Using Bitmaps. In Proceedings of the Eigth ACM SIGKDD International Conference of Knowledge Discovery and Data Mining. 429–435.
- Jichen Zhu and Santiago Ontañón . 2020 . Player-Centered AI for Automatic Game Personalization: Open Problems. In International Conference on the Foundations of Digital Games. 1–8. Jichen Zhu and Santiago Ontañón. 2020. Player-Centered AI for Automatic Game Personalization: Open Problems. In International Conference on the Foundations of Digital Games. 1–8.
Cited by
Alghamdi, E., Halvey, M., & Nicol, E. (2024) · ACM Conversational User Interfaces 2024 · doi:10.1145/3640794.3665558 · Cited by 36
Glasser, A., Watkins, M., Hart, K., Lee, S., & Huenerfauth, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501987 · Cited by 22
Conversations with the News: Co-speculation into Conversational Interactions with News Content
Nordberg, O. E., & Guribye, F. (2023) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3571884.3597123 · Cited by 14
“These troubles are usually answered by making a new utterance, by hyperarticulating or simplifying the utterance, adding more information, relying on graphical user interface, a frustration attempt, recall, or by the person settling, restarting, or quitting the interaction [36].”
BackgroundLaw, E. L., van As, N., & Følstad, A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42283-6_16 · Cited by 12
Vianny, M. M., Ayyaswamy, K., Rithik, G., Kathirvel, N., & Anthony Raj, A. (2025) · Advances in medical technologies and clinical practice book series · doi:10.4018/979-8-3693-7550-1.ch015 · Cited by 5
Metadata
- Authors
- Myers, C. M., Laris Pardo, L. F., Acosta-Ruiz, A., Canossa, A., & Zhu, J.
- Affiliations
- Drexel University (US)
- Royal Danish Academy of Fine Arts, Denmark
- Venue
- CUI 2021 - 3rd Conference on Conversational User Interfaces
- Publisher
- ACM
- Published
- 2021-07-27
- Pages
- 1-8
- OpenAlex
- W3186527222
- Cited by
- 10
- References
- 38
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 2 · 2023: 2 · 2022: 1
- FWCI
- 0.58
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Conversational User Interfaces (CUI) (publishedVersion)
- IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion)
- Architecture, Design and Conservation (Aarhus School of Architecture, Design School Kolding, The Royal Danish Academy of Fine Arts, Schools of Architecture, Design and Conservation (KADK)) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- voice user interface, user interaction patterns, sequence analysis, trial-and-error learning, utterance structure, entity recognition, multimodal interfaces, user behavior clustering
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: “Try, Try, Try Again:” Sequence Analysis of User Interaction Data with a Voice User Interface. https://app.cochat.ai/papers/10.1145/3469595.3469613
- The paper
- Myers, C. M., Laris Pardo, L. F., Acosta-Ruiz, A., Canossa, A., & Zhu, J. (2021). “Try, Try, Try Again:” Sequence Analysis of User Interaction Data with a Voice User Interface. CUI 2021 - 3rd Conference on Conversational User Interfaces. https://doi.org/10.1145/3469595.3469613
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.