CoChat

“Try, Try, Try Again:” Sequence Analysis of User Interaction Data with a Voice User Interface

Myers, Chelsea M., Laris Pardo, Luis Fernando, Acosta-Ruiz, Ana, Canossa, Alessandro, Zhu, Jichen · 2021 · CUI 2021 - 3rd Conference on Conversational User Interfaces

The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.

Abstract

Voice User Interfaces (VUIs) pose challenges for users to learn the system’s supported features and commands. Users often rely on trial-and-error to navigate VUI dialogues and complete desired tasks. The order in which users try different commands contains vital information about how they learn. In this paper, we explore using sequence analysis techniques to reveal the patterns of tactics our participants (n = 50) used when interacting with an unfamiliar multi-modal VUI. We present the results of our sequence analysis, the top sequences used, and a cluster analysis of our participants on their usage of the top sequences. Our results indicate participants initially struggled with understanding the acceptable utterance structure and entities more so than utterance keywords. Additionally, we discuss our participants’ behavior differences and discuss the usefulness of this methodology of sequence analysis for HCI VUI research.

Overview

Integrity signals

Registry

Crossref · 10.1145/3469595.3469613

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 38 references

Research works printed but not linked: 12

  1. John Brooke . 1996. SUS A quick and dirty usability scale. Usability evaluation in industry 189, 194 ( 1996 ), 4–7. John Brooke. 1996. SUS A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7.
  2. William Jay Conover and Ronald L Iman . 1979 . On multiple-comparisons procedures. Los Alamos Sci. Lab. Tech. Rep. LA-7677-MS 1 (1979), 14 . William Jay Conover and Ronald L Iman. 1979. On multiple-comparisons procedures. Los Alamos Sci. Lab. Tech. Rep. LA-7677-MS 1 (1979), 14.
  3. 10.1145/3332167.3357103 · DOI
  4. Kristiina Jokinen , Kari Kanto , Antti Kerminen , and Jyrki Rissanen . 2004 . Evaluation of Adaptivity and User Expertise in a Speech-Based E-Mail System . Proc. of the 20th ACL International Conference on Computational LinguisticMarch 2016(2004) , 44–52. Kristiina Jokinen, Kari Kanto, Antti Kerminen, and Jyrki Rissanen. 2004. Evaluation of Adaptivity and User Expertise in a Speech-Based E-Mail System. Proc. of the 20th ACL International Conference on Computational LinguisticMarch 2016(2004), 44–52.
  5. Pavan Kantharaju , Katelyn Alderfer , Jichen Zhu , Bruce Char , Brian Smith , and Santiago Ontanón . 2018 . Tracing player knowledge in a parallel programming educational game . In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 14 . Pavan Kantharaju, Katelyn Alderfer, Jichen Zhu, Bruce Char, Brian Smith, and Santiago Ontanón. 2018. Tracing player knowledge in a parallel programming educational game. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 14.
  6. Sasha Makarovych , Alessandro Canossa , Julian Togelius , and Anders Drachen . 2018 . Like a DNA string: Sequence-based player profiling in Tom Clancy’s The Division. In Artificial Intelligence and Interactive Digital Entertainment Conference. York. Sasha Makarovych, Alessandro Canossa, Julian Togelius, and Anders Drachen. 2018. Like a DNA string: Sequence-based player profiling in Tom Clancy’s The Division. In Artificial Intelligence and Interactive Digital Entertainment Conference. York.
  7. Martin Porcheron , Joel E. Fischer , Stuart Reeves , and Sarah Sharples . 2018. Voice Interfaces in Everyday Life. CHI ’18 ( 2018 ), 1–12. https://doi.org/doi.org/10.1145/3173574.3174214 Martin Porcheron, Joel E. Fischer, Stuart Reeves, and Sarah Sharples. 2018. Voice Interfaces in Everyday Life. CHI ’18 (2018), 1–12. https://doi.org/doi.org/10.1145/3173574.3174214 · DOI · Paper page
  8. Rafet Sifa , Anders Drachen , and Christian Bauckhage . 2018. Profiling in games: Understanding behavior from telemetry. Social interactions in virtual worlds: An interdisciplinary perspective ( 2018 ). Rafet Sifa, Anders Drachen, and Christian Bauckhage. 2018. Profiling in games: Understanding behavior from telemetry. Social interactions in virtual worlds: An interdisciplinary perspective (2018).
  9. Stefan Ultes , Robert ElChab , and Wolfgang Minker . 2014. Application and evaluation of a conditioned hidden markov model for estimating interaction quality of spoken dialogue systems . In Natural Interaction with Robots, Knowbots and Smartphones . Springer , 303–312. Stefan Ultes, Robert ElChab, and Wolfgang Minker. 2014. Application and evaluation of a conditioned hidden markov model for estimating interaction quality of spoken dialogue systems. In Natural Interaction with Robots, Knowbots and Smartphones. Springer, 303–312.
  10. Josep Valls-Vargas , Santiago Ontanón , and Jichen Zhu . 2015 . Exploring player trace segmentation for dynamic play style prediction . In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 11 . Josep Valls-Vargas, Santiago Ontanón, and Jichen Zhu. 2015. Exploring player trace segmentation for dynamic play style prediction. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Vol. 11.
  11. G Wallner . 2002 . Sequential Pattern Mining Using Bitmaps . In Proceedings of the Eigth ACM SIGKDD International Conference of Knowledge Discovery and Data Mining. 429–435 . G Wallner. 2002. Sequential Pattern Mining Using Bitmaps. In Proceedings of the Eigth ACM SIGKDD International Conference of Knowledge Discovery and Data Mining. 429–435.
  12. Jichen Zhu and Santiago Ontañón . 2020 . Player-Centered AI for Automatic Game Personalization: Open Problems. In International Conference on the Foundations of Digital Games. 1–8. Jichen Zhu and Santiago Ontañón. 2020. Player-Centered AI for Automatic Game Personalization: Open Problems. In International Conference on the Foundations of Digital Games. 1–8.

Cited by

See all 11

Metadata

Authors
Myers, C. M., Laris Pardo, L. F., Acosta-Ruiz, A., Canossa, A., & Zhu, J.
Affiliations
  • Drexel University (US)
  • Royal Danish Academy of Fine Arts, Denmark
Venue
CUI 2021 - 3rd Conference on Conversational User Interfaces
Publisher
ACM
Published
2021-07-27
Pages
1-8
OpenAlex
W3186527222
Cited by
10
References
38
Citations by year
2026: 2 · 2025: 3 · 2024: 2 · 2023: 2 · 2022: 1
FWCI
0.58
Type
conference paper
Language
en
Open access
closed
Locations
  • ACM Conference on Conversational User Interfaces (CUI) (publishedVersion)
  • IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion)
  • Architecture, Design and Conservation (Aarhus School of Architecture, Design School Kolding, The Royal Danish Academy of Fine Arts, Schools of Architecture, Design and Conservation (KADK)) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
Keywords
voice user interface, user interaction patterns, sequence analysis, trial-and-error learning, utterance structure, entity recognition, multimodal interfaces, user behavior clustering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: “Try, Try, Try Again:” Sequence Analysis of User Interaction Data with a Voice User Interface. https://app.cochat.ai/papers/10.1145/3469595.3469613
The paper
Myers, C. M., Laris Pardo, L. F., Acosta-Ruiz, A., Canossa, A., & Zhu, J. (2021). “Try, Try, Try Again:” Sequence Analysis of User Interaction Data with a Voice User Interface. CUI 2021 - 3rd Conference on Conversational User Interfaces. https://doi.org/10.1145/3469595.3469613

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.