Understanding and Measuring User Experience in Conversational Interfaces
Abstract
Although various methods have been developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This paper reviews the understandings of user experience (UX) in conversational interfaces literature and examines the six questionnaires commonly used for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure different UX dimensions in that context. The method to examine the questionnaires involved developing an assessment framework for main UX dimensions with relevant attributes and coding the items in the questionnaires according to the framework. The results show that (i) the understandings of UX notably differed in literature; (ii) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (iii) while the dimension of affect was covered by two questionnaires, playfulness, motivation, and frustration dimensions were covered by one questionnaire only. The largest coverage of UX dimensions has been provided by the Subjective Assessment of Speech System Interfaces (SASSI). We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension.RESEARCH HIGHLIGHTS: Varying understandings of UX in conversational interfaces literature. A UX assessment framework with UX dimensions and their relevant attributes. Descriptions of the six main questionnaires for evaluating conversational interfaces. A comparison of the six questionnaires based on their coverage of UX dimensions.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Evaluation in chatbot research is conducted through instruments for users' self reports of user experience, user observation and interviews, analyses of chatbot interaction, and physiological measurements.
“Evaluation in chatbot research is conducted by instruments for users' self reports of user experience [63], through user observation and interviews [75] and analyses of chatbot interaction [66], and also by physiological measurements [22].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [63]
There is a lack of common definitions, metrics, and validated scales for key aspects of chatbot evaluations.
“While there is a large number of studies on chatbot user experience available, there is a lack of common definitions, metrics and validated scales for key aspects of chatbot evaluations [63].”
From Future directions for chatbot research: An interdisciplinary research agenda (2021) · cites this paper as [63]
Cites
Measuring emotion: The self-assessment manikin and the semantic differential
Bradley, M. M., & Lang, P. J. (1994) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(94)90063-9 · Cited by 9,538 · see citing works
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989 · see citing works
(1974) · Medical Entomology and Zoology · Cited by 7,295 · see citing works
DEWEY, J. M. (2026) · State University of New York Press eBooks · doi:10.1515/9798855805482-007 · Cited by 5,188 · see citing works
Emotional design: why we love (or hate) everyday things
(2004) · Choice Reviews Online · doi:10.5860/choice.41-6846 · Cited by 3,086 · see citing works
McAuley, E., Duncan, T., & Tammen, V. V. (1989) · Research Quarterly for Exercise and Sport · doi:10.1080/02701367.1989.10607413 · Cited by 2,694 · see citing works
Content validity in psychological assessment: A functional approach to concepts and methods.
Haynes, S. N., Richard, D. C. S., & Kubany, E. S. (1995) · Psychological Assessment · doi:10.1037//1040-3590.7.3.238 · Cited by 2,387 · see citing works
Development and Validation of a Scale to Measure Optimal Experience: The Flow State Scale
Jackson, S. A., & Marsh, H. W. (1996) · Journal of Sport and Exercise Psychology · doi:10.1123/jsep.18.1.17 · Cited by 1,567 · see citing works
What is user engagement? A conceptual framework for defining user engagement with technology
O'Brien, H. L., & Toms, E. G. (2008) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.20801 · Cited by 1,397 · see citing works
McCarthy, J., & Wright, P. (2004) · The MIT Press eBooks · doi:10.7551/mitpress/6687.003.0011 · Cited by 1,371 · see citing works
Assessing dimensions of perceived visual aesthetics of web sites
Lavie, T., & Tractinsky, N. (2003) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2003.09.002 · Cited by 1,349 · see citing works
Current practice in measuring usability: Challenges to usability studies and research
Hornbæk, K. (2006) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2005.06.002 · Cited by 1,123 · see citing works
The dimensionality and correlates of flow in human-computer interactions
Webster, J., Treviño, L. K., & Ryan, L. (1993) · Computers in Human Behavior · doi:10.1016/0747-5632(93)90032-n · Cited by 1,103 · see citing works
McCarthy, J., & Wright, P. R. (2004) · interactions · doi:10.1145/1015530.1015549 · Cited by 1,015 · see citing works
AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität
Hassenzahl, M., Burmester, M., & Koller, F. (2003) · Berichte des German Chapter of the ACM · doi:10.1007/978-3-322-80058-9_19 · Cited by 955 · see citing works
Norman, D. A. (2004) · Ubiquity · doi:10.1145/985600.966013 · Cited by 851 · see citing works
What Should We Do about Motivation Theory? Six Recommendations for the Twenty-First Century
Locke, E. A., & Latham, G. P. (2004) · Academy of Management Review · doi:10.5465/amr.2004.13670974 · Cited by 838 · see citing works
McTear, M., Callejas, Z., & Griol, D. (2016) · doi:10.1007/978-3-319-32967-3 · Cited by 594 · see citing works
The Effect of Perceived Hedonic Quality on Product Appealingness
Hassenzahl, M. (2001) · International Journal of Human-Computer Interaction · doi:10.1207/s15327590ijhc1304_07 · Cited by 487 · see citing works
Lee, S., & Choi, J. (2017) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2017.02.005 · Cited by 388 · see citing works
Towards a tool for the Subjective Assessment of Speech System Interfaces (SASSI)
Hone, K., & Graham, R. (2000) · Natural Language Engineering · doi:10.1017/s1351324900002497 · Cited by 355 · see citing works
The Inference of Perceived Usability From Beauty
Hassenzahl, M., & Monk, A. F. (2010) · Human-Computer Interaction · doi:10.1080/07370024.2010.500139 · Cited by 331 · see citing works
Co-experience: user experience as interaction
Battarbee, K., & Koskinen, I. (2004) · CoDesign · doi:10.1080/15710880412331289917 · Cited by 313 · see citing works
The Adult Playfulness Scale: An Initial Assessment
Glynn, M. A., & Webster, J. (1992) · Psychological Reports · doi:10.2466/pr0.1992.71.1.83 · Cited by 282 · see citing works
Lallemand, C., Gronier, G., & Koenig, V. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.10.048 · Cited by 248 · see citing works
The experience of enchantment in human–computer interaction
McCarthy, J., Wright, P., Wallace, J., & Dearden, A. (2005) · Personal and Ubiquitous Computing · doi:10.1007/s00779-005-0055-2 · Cited by 173 · see citing works
Hassenzahl, M., Schöbel, M., & Trautmann, T. (2008) · Interacting with Computers · doi:10.1016/j.intcom.2008.05.001 · Cited by 129 · see citing works
Evaluation and usability of multimodal spoken language dialogue systems
Dybkjær, L., Bernsen, N. O., & Minker, W. (2004) · Speech Communication · doi:10.1016/j.specom.2004.02.001 · Cited by 118 · see citing works
Reliability, validity, and sensitivity of a single-item measure of online store usability
Christophersen, T., & Konradt, U. (2010) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2010.10.005 · Cited by 104 · see citing works
Syrdal, A. K., Bennett, R., & Greenspan, S. (1995) · CRC Press eBooks · Cited by 79 · see citing works
Blythe, M., Reid, J., Wright, P., & Geelhoed, E. (2006) · Behaviour and Information Technology · doi:10.1080/01449290500331131 · Cited by 68 · see citing works
Vanderdonckt, J. M. (2014) · Routledge eBooks · doi:10.4324/9781410607409 · Cited by 54 · see citing works
Expanding the MOS: Development and Psychometric Evaluation of the MOS-R and MOS-X
Polkosky, M. D., & Lewis, J. R. (2003) · International Journal of Speech Technology · doi:10.1023/a:1022390615396 · Cited by 51 · see citing works
Measuring the Quality of Service and Quality of Experience of multimodal human–machine interaction
Wechsung, I., Engelbrecht, K.-P., Kühnel, C., Möller, S., & Weiß, B. (2012) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-011-0088-y · Cited by 50 · see citing works
Designing habitable dialogues for speech-based interaction with computers
Hone, K., & Baber, C. (2001) · Int. J. Hum. Comput. Stud. · doi:10.1006/ijhc.2000.0456 · Cited by 44 · see citing works
Rajanen, D., Clemmensen, T., Iivari, N., İnal, Y., Rızvanoğlu, K., Sivaji, A., & Roche, A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-68059-0_14 · Cited by 38 · see citing works
Investigating the psychometric properties of the Speech User Interface Service Quality questionnaire
Lewis, J. R., & Hardzinski, M. L. (2015) · International Journal of Speech Technology · doi:10.1007/s10772-015-9289-1 · Cited by 37 · see citing works
Quantifying Quality Aspects of Multimodal Interactive Systems
Kühnel, C. (2012) · T-labs series in telecommunication services · doi:10.1007/978-3-642-29602-4 · Cited by 19 · see citing works
A Psychology of User Experience
Turner, P. (2017) · Human-computer interaction series · doi:10.1007/978-3-319-70653-5 · Cited by 19 · see citing works
Enhancement of Conversational Agents By Means of Multimodal Interaction
López-Cózar, R., Callejas, Z., Espejo, G., & Griol, D. (2011) · Conversational Agents and Natural Language Interaction · doi:10.4018/978-1-60960-617-6.ch010 · Cited by 18 · see citing works
A Psychology of User Experience: Involvement, Affect and Aesthetics
Turner, P. (2017) · CERN Document Server (European Organization for Nuclear Research) · Cited by 15 · see citing works
Human-Computer Interaction - INTERACT 2017
Bernhaupt, R., Dalvi, G., Joshi, A., et al. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67684-5 · Cited by 5 · see citing works
Quantifying Quality Aspects of a Multimodal Interactive System
Kühnel, C. (2012) · T-labs series in telecommunication services · doi:10.1007/978-3-642-29602-4_7 · Cited by 3 · see citing works
How to Evaluate? Development of the MMQQ (MultiModal Quality Questionnaire)
Wechsung, I. (2014) · T-labs series in telecommunication services · doi:10.1007/978-3-319-03810-0_4 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Future directions for chatbot research: An interdisciplinary research agenda
Følstad, A., Araujo, T., Law, E. L.-C., Brandtzaeg, P. B., Papadopoulos, S., Reis, L., Baez, M., Laban, G., McAllister, P., Ischen, C., et al. (2021) · Computing · doi:10.1007/s00607-021-01016-7 · Cited by 247 · see citing works
Measuring Voice UX Quantitatively: A Rapid Review
Seaborn, K., & Urakami, J. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451712 · Cited by 55 · see citing works
Iniesto, F., Coughlan, T., Lister, K., Devine, P., Freear, N., Greenwood, R., Holmes, W., Kenny, I., McLeod, K., & Tudor, R. (2023) · ACM Transactions on Accessible Computing · doi:10.1145/3568166 · Cited by 25 · see citing works
Chen, Z., Xiong, Z., Ruan, X., Jiang, S., Wei, W., & Fang, K. (2024) · 2024 4th International Conference on Educational Technology (ICET) · doi:10.1109/icet62460.2024.10868064 · Cited by 7 · see citing works
- Authors
- Kocaballi, A. B., Laranjo, L., & Coiera, E.
- Affiliations
- Macquarie University (AU)
- Corresponding author
- Kocaballi, Ahmet Baki
- Venue
- Interacting with Computers
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-03-01
- Volume
- 31
- Issue
- 2
- Pages
- 192-207
- ISSN
- 0953-5438 (ISSN-L)
- 1873-7951
- OpenAlex
- W2945452260
- Cited by
- 68
- References
- 44
- Citations by year
- 2026: 5 · 2025: 10 · 2024: 14 · 2023: 9 · 2022: 13 · 2021: 10 · 2020: 7
- FWCI
- 7.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Interacting with Computers (publishedVersion)
- OPUS - Open Publications of UTS Scholars (University of Technology Sydney) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Artificial Intelligence › AI in Service Interactions
- Keywords
- user experience, conversational interfaces, SASSI, affect, playfulness, motivation, frustration, evaluation framework
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
