Wizard of Oz Experimentation for Language Technology Applications: Challenges and Tools
Open accessThe DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
Wizard of OZ (WOZ) is a well-established method for simulating the functionality and user experience of future systems. Using a human wizard to mimic certain operations of a potential system is particularly useful in situations where extensive engineering effort would otherwise be needed to explore the design possibilities offered by such operations. The WOZ method has been widely used in connection with speech and language technologies, but advances in sensor technology and pattern recognition as well as new application areas such as human-robot interaction have made it increasingly relevant to the design of a wider range of interactive systems. In such cases achieving acceptable performance at the user interface level often hinges on resource intensive improvements such as domain tuning, which are better done once the overall design is relatively stable. While WOZ is recognised as a valuable prototyping technique, surprisingly little effort has been put into exploring it from a methodological point of view. Starting from a survey of the literature, this paper presents a systematic investigation and analysis of the design space for WOZ for language technology applications, and proposes a generic architecture for tool support that supports the integration of components for speech recognition and synthesis as well as for machine translation. This architecture is instantiated in WebWOZ - a new web-based open-source WOZ prototyping platform. The viability of generic support is explored empirically through a series of evaluations. Researchers from a variety of backgrounds were able to create experiments, independent of their previous experience with WOZ. The approach was further validated through a number of real experiments, which also helped to identify a number of possibilities for additional support, and flagged potential issues relating to consistency in Wizard performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989
Suede: a Wizard of Oz prototyping tool for speech user interfaces
Klemmer, S. R., Sinha, A. K., Chen, J., Landay, J. A., Aboobaker, N., & Wang, A. (2000) · Proceedings of the 13th annual ACM symposium on User interface software and technology - UIST '00 · doi:10.1145/354401.354406 · Cited by 230
Cited by
Feedpal: Understanding Opportunities for Chatbots in Breastfeeding Education of Women in India
Yadav, D., Malik, P., Dabas, K., & Singh, P. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359272 · Cited by 90
Planning for Natural Language Failures with the AI Playbook
Hong, M. K., Fourney, A., DeBellis, D., & Amershi, S. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445735 · Cited by 54
“WoZ techniques were originally developed in the context of envisioning future intelligent language experiences [12] and have evolved over the course of three decades to aid wizards in tasks such as simulating realistic system responses and reducing response latency [25, 26].”
BackgroundMethodAssessing Voice User Interfaces: The vassist system prototype
Cordasco, G., Esposito, M., Masucci, F., Riviello, M. T., Esposito, A., Chollet, G., Schlogl, S., Milhorat, P., & Pelosi, G. (2014) · 2014 5th IEEE Conference on Cognitive Infocommunications (CogInfoCom) · doi:10.1109/coginfocom.2014.7020425 · Cited by 28
“In addition, a second laptop was connected to the same wireless router so that a Wizard of Oz web application [10] deployed on the vAssist system was available on this second machine.”
MethodAgha, Z., Park, J., Wan, R., Ali, N. S., Wang, Y., Difranzo, D., Badillo-Urquiola, K., & Wisniewski, P. J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642313 · Cited by 13
“• Maintain a simple choice architecture by using follow-up nudges and confirmation prompts as ways to present multiple options in a minimized way.”
Background
Metadata
- Authors
- Schlögl, S., Doherty, G., & Luz, S.
- Venue
- Interacting with Computers
- Publisher
- Oxford University Press (OUP)
- Published
- 2014-05-09
- Volume
- 27
- Issue
- 6
- Pages
- 592-615
- ISSN
- 0953-5438 (ISSN-L)
- 1873-7951
- OpenAlex
- W2108687163
- Cited by
- 43
- References
- 2
- FWCI
- 2.64
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Wizard of Oz Experimentation for Language Technology Applications: Challenges and Tools. https://app.cochat.ai/papers/10.1093/iwc/iwu016
- The paper
- Schlögl, S., Doherty, G., & Luz, S. (2015). Wizard of Oz Experimentation for Language Technology Applications: Challenges and Tools. Interacting with Computers. https://doi.org/10.1093/iwc/iwu016
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.