The Chatbot Usability Scale: the Design and Pilot of a Usability Scale for Interaction with AI-Based Conversational Agents
Open accessThe DOI matches its Crossref record; 67 of the 68 references it cites were found in a registry or an index.
Abstract
Standardised tools to assess a user’s satisfaction with the experience of using chatbots and conversational agents are currently unavailable. This work describes four studies, including a systematic literature review, with an overall sample of 141 participants in the survey (experts and novices), focus group sessions and testing of chatbots to (i) define attributes to assess the quality of interaction with chatbots and (ii) the designing and piloting a new scale to measure satisfaction after the experience with chatbots. Two instruments were developed: (i) A diagnostic tool in the form of a checklist (BOT-Check). This tool is a development of previous works which can be used reliably to check the quality of a chatbots experience in line with commonplace principles. (ii) A 15-item questionnaire (BOT Usability Scale, BUS-15) with estimated reliability between .76 and .87 distributed in five factors. BUS-15 strongly correlates with UMUX-LITE by enabling designers to consider a broader range of aspects usually not considered in satisfaction tools for non-conversational agents, e.g. conversational efficiency and accessibility, quality of the chatbot’s functionality and so on. Despite the convincing psychometric properties, BUS-15 requires further testing and validation. Designers can use it as a tool to assess products, thus building independent databases for future evaluation of its reliability, validity and sensitivity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989
Usability Evaluation In Industry
(1996) · CRC Press eBooks · doi:10.1201/9781498710411 · Cited by 4,354
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · cited 2×
“The agreement among the participants was equal to 0.780, which is acceptably higher than the minimum level of .667 for the Krippendorff ’ s Alpha [35].”
MethodResultInfluentialFitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730
The measurement of user information satisfaction
Ives, B., Olson, M. H., & Baroudi, J. J. (1983) · Communications of the ACM · doi:10.1145/358413.358430 · Cited by 1,820
Service loyalty: The effects of service quality and the mediating role of customer satisfaction
Caruana, A. (2002) · European Journal of Marketing · doi:10.1108/03090560210430818 · Cited by 1,528
Trust in a specific technology: An investigation of its components and measures
Mcknight, D. H., Carter, M., Thatcher, J. B., & Clay, P. F. (2011) · ACM Transactions on Management Information Systems · doi:10.1145/1985347.1985353 · Cited by 1,185
Go, E., & Sundar, S. S. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.01.020 · Cited by 1,089
Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape
Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S., & Torous, J. B. (2019) · The Canadian Journal of Psychiatry · doi:10.1177/0706743719828977 · Cited by 1,048
Chatbot e-service and customer satisfaction regarding luxury brands
Chung, M., Ko, E., Joung, H., & Kim, S. J. (2020) · Journal of Business Research · doi:10.1016/j.jbusres.2018.10.004 · Cited by 998
Brandtzaeg, P. B., & Følstad, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-70284-1_30 · Cited by 788
The Semiotic Engineering of Human-Computer Interaction
de Souza, C. S. (2005) · doi:10.7551/mitpress/6175.001.0001 · Cited by 757 · cited 1×
“The ability to communicate and maintain an efficient and effective conversational exchange is not a secondary, but actually, a characterising element of chatbots that should be considered in the assessment of satisfaction with these tools [18, 19].”
BackgroundMeasuring usability: are effectiveness, efficiency, and satisfaction really correlated?
Frøkjær, E., Hertzum, M., & Hornbæk, K. (2000) · Proceedings of the SIGCHI conference on Human Factors in Computing Systems · doi:10.1145/332040.332455 · Cited by 650
The Usability Metric for User Experience
Finstad, K. (2010) · Interacting with Computers · doi:10.1016/j.intcom.2010.04.004 · Cited by 604
Chaves, A. P., & Gerosa, M. A. (2021) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1841438 · Cited by 572
Chatbots and the new world of HCI
Følstad, A., & Brandtzæg, P. B. (2017) · Interactions · doi:10.1145/3085558 · Cited by 513
DALE, R. (2016) · Natural Language Engineering · doi:10.1017/s1351324916000243 · Cited by 485
Verhagen, T., van Nes, J., Feldberg, F., & van Dolen, W. (2014) · Journal of Computer-Mediated Communication · doi:10.1111/jcc4.12066 · Cited by 403
Lee, S., & Choi, J. (2017) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2017.02.005 · Cited by 388
Evaluating and Informing the Design of Chatbots
Jain, M., Kumar, P., Kota, R., & Patel, S. N. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196735 · Cited by 376
Towards Designing Cooperative and Social Conversational Agents for Customer Service
Gnewuch, U., Morana, S., & Mädche, A. (2017) · AIS Electronic Library (AISeL) · Cited by 372
Miner, A. S., Milstein, A., Schueller, S., Hegde, R., Mangurian, C., & Linos, E. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.0400 · Cited by 368
Chatbots: changing user needs and motivations
Brandtzaeg, P. B., & Følstad, A. (2018) · Interactions · doi:10.1145/3236669 · Cited by 322
Towards a tool for the Subjective Assessment of Speech System Interfaces (SASSI)
HONE, K. S., & GRAHAM, R. (2000) · Natural Language Engineering · doi:10.1017/s1351324900002497 · Cited by 305
UMUX-LITE: when there's no time for the SUS
Lewis, J. R., Utesch, B. S., & Maher, D. E. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2481287 · Cited by 270
Chung, H., Iorga, M., Voas, J., & Lee, S. (2017) · Computer · doi:10.1109/mc.2017.3571053 · Cited by 230
Semiotic Engineering Methods for Scientific Research in HCI
de Souza, C. S., & Leitão, C. F. (2009) · Synthesis lectures on human-centered informatics · doi:10.2200/s00173ed1v01y200901hci002 · Cited by 209
Bayesian exploratory factor analysis
Conti, G., Frühwirth‐Schnatter, S., Heckman, J. J., & Piatek, R. (2014) · Journal of Econometrics · doi:10.1016/j.jeconom.2014.06.008 · Cited by 183
Borsci, S., Federici, S., Bacci, S., Gnaldi, M., & Bartolucci, F. (2015) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2015.1064648 · Cited by 180
Self-citationFølstad, A., Skjuve, M. B., & Brandtzæg, P. B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-17705-8_13 · Cited by 173
I'm Sorry, Dave, I'm Afraid I Can't Do That: Chatbot Perception and Expectations
Zamora, J. (2017) · Proceedings of the 5th International Conference on Human Agent Interaction · doi:10.1145/3125739.3125766 · Cited by 164
Trust in information sources: Seeking information from people, documents, and virtual agents
Hertzum, M., Andersen, H. H. K., Andersen, V., & Hansen, C. B. (2002) · Interacting with Computers · doi:10.1016/s0953-5438(02)00023-1 · Cited by 157
PARADISE: a framework for evaluating spoken dialogue agents
Walker, M. A., Litman, D. J., Kamm, C. A., & Abella, A. (1997) · Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics - · doi:10.3115/979617.979652 · Cited by 138
Chatbots and conversational agents: A bibliometric analysis
Io, H. N., & Lee, C. B. (2017) · 2017 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM) · doi:10.1109/ieem.2017.8289883 · Cited by 137
Hoofs, H., van de Schoot, R., Jansen, N., & Kant, I. (2017) · Educational and Psychological Measurement · doi:10.1177/0013164417709314 · Cited by 127
The linguistic accuracy of chatbots: usability from an ESL perspective
Coniam, D. (2014) · Text & Talk · doi:10.1515/text-2014-0018 · Cited by 121
Designing an interactive open-domain question answering system
Quarteroni, S., & Manandhar, S. (2008) · Natural Language Engineering · doi:10.1017/s1351324908004919 · Cited by 113
‘Realness’ in Chatbots: Establishing Quantifiable Criteria
Morrissey, K., & Kirakowski, J. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39330-3_10 · Cited by 110
Kuligowska, K. (2015) · Professionals Center for Business Research · doi:10.18483/pcbr.22 · Cited by 97
A framework for understanding chatbots and their future
Paikari, E., & van der Hoek, A. (2018) · Proceedings of the 11th International Workshop on Cooperative and Human Aspects of Software Engineering · doi:10.1145/3195836.3195859 · Cited by 67
Multi-tasking and collaborative activities in dialogue systems
Lemon, O., Gruenstein, A., Battle, A., & Peters, S. (2002) · Proceedings of the 3rd SIGdial workshop on Discourse and dialogue - · doi:10.3115/1118121.1118137 · Cited by 53
New technologies and mixed-use convergence: How humans and algorithms are adapting to each other
Applin, S. A., & Fischer, M. D. (2015) · 2015 IEEE International Symposium on Technology and Society (ISTAS) · doi:10.1109/istas.2015.7439436 · Cited by 51
Is Your Company Ready to Operate as a Market?
McGrath, R. G. (2018) · What the Digital Future Holds · doi:10.7551/mitpress/11645.003.0011 · Cited by 50
Detecting Deceptive Chat-Based Communication Using Typing Behavior and Message Cues
Derrick, D. C., Meservy, T. O., Jenkins, J. L., Burgoon, J. K., & Nunamaker, J. F. (2013) · ACM Transactions on Management Information Systems · doi:10.1145/2499962.2499967 · Cited by 48
Federici, S., Filippis, M. L. D., Mele, M. L., Borsci, S., Bracalenti, M., Gaudino, G., Cocco, A., Amendola, M., & Simonetti, E. (2020) · Disability and Rehabilitation Assistive Technology · doi:10.1080/17483107.2020.1775313 · Cited by 44
Self-citationHere's What I Can Do: Chatbots' Strategies to Convey Their Features to Users
Valério, F. A. M., Guimarães, T. G., Prates, R. O., & Candello, H. (2017) · Proceedings of the XVI Brazilian Symposium on Human Factors in Computing Systems · doi:10.1145/3160504.3160544 · Cited by 42
Measuring User Experience With 3, 5, 7, or 11 Points: Does It Matter?
Lewis, J. R. (2019) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720819881312 · Cited by 40
Investigating the psychometric properties of the Speech User Interface Service Quality questionnaire
Lewis, J. R., & Hardzinski, M. L. (2015) · International Journal of Speech Technology · doi:10.1007/s10772-015-9289-1 · Cited by 39
Chatbots for troubleshooting: A survey
Thorne, C. (2017) · Language and Linguistics Compass · doi:10.1111/lnc3.12253 · Cited by 36
User Satisfaction, Aesthetics and Usability
Lindgaard, G., & Dudek, C. (2002) · IFIP advances in information and communication technology · doi:10.1007/978-0-387-35610-5_16 · Cited by 31
Exploring expressivity and emotion with artificial voice and speech technologies
Pauletto, S., Balentine, B., Pidcock, C. J., Jones, K., Bottaci, L., Aretoulaki, M., Wells, J., Mundy, D. P., & Balentine, J. (2013) · Logopedics Phoniatrics Vocology · doi:10.3109/14015439.2013.810303 · Cited by 28
Polkosky, M. D. (2008) · Routledge eBooks · doi:10.4324/9780203926864-11 · Cited by 24
Establishing the Hallmarks of a Convincing Chatbot-Human Dialogue
Kirakowski, J., ODonnell, P., & Yiu, A. (2009) · InTech eBooks · doi:10.5772/7741 · Cited by 23
Switching to real-time tasks in multi-tasking dialogue
Yang, F., Heeman, P. A., & Kun, A. L. (2008) · International Conference on Computational Linguistics · doi:10.3115/1599081.1599210 · Cited by 17
Vetter, M. (2002) · Springer eBooks · doi:10.1007/978-3-642-56333-1_11 · Cited by 12
My phone assistant should know I am an Indian
Chopra, S., & Chivukula, S. S. (2017) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/3098279.3122137 · Cited by 10
ALife-WebGuide: an intelligent user interface for Web site navigation
Gaudiano, P., & Kater, K. (2000) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/325737.325800 · Cited by 8
EmotionPush: Emotion and Response Time Prediction Towards Human-Like Chatbots
Huang, C.-Y., & Ku, L. (2018) · Globecom · doi:10.1109/glocom.2018.8647331 · Cited by 8
Models towards a hybrid conversational agent for contact centres
Sankar, G. R., Greyling, J., Vogts, D., & du Plessis, M. C. (2008) · South African Institute of Computer Scientists and Information Technologists · doi:10.1145/1456659.1456683 · Cited by 7
An Oral Exam for Measuring a Dialog System’s Capabilities
Cohen, D., & Lane, I. R. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10060 · Cited by 6
McTear, M., Callejas, Z., & Griol, D. (2016) · Springer eBooks · doi:10.1007/978-3-319-32967-3_5 · Cited by 6
BayesFM: Bayesian Inference for Factor Modeling
Piatek, R. (2016) · doi:10.32614/cran.package.bayesfm · Cited by 6
Weiss, A., & Tscheligi, M. (2012) · Springer eBooks · doi:10.1007/978-3-642-32323-2_1 · Cited by 5
Pereira, J., & Díaz, Ó. (2019) · Lecture notes in business information processing · doi:10.1007/978-3-030-28430-5_4 · Cited by 5
Hinrichs, H., & Le, N.-T. (2018) · Electronic workshops in computing · doi:10.14236/ewic/hci2018.211 · Cited by 4
Research works printed but not linked: 13
- Charlton JI (2000) Nothing about us without us: Disability oppression and empowerment. Univ of California Press
- DeVellis, R. F. (2016). Scale development: Theory and applications (Vol. 26). Sage publications.
- Dillon A (2001) Beyond Usability: Process, Outcome and Affect in human computer interactions. Canadian Journal of Information and Library Science 26(4)
- Feine, J., Morana, S., & Gnewuch, U. (2019). Measuring service encounter satisfaction with customer service chatbots using sentiment analysis 14th International Conference on Wirtschaftsinformatik (WI2019), Siegen, Germany. https://aisel.aisnet.org/wi2019/
- Harkous, H., Fawaz, K., Shin, K. G., & Aberer, K. (2016). Pribots: Conversational privacy with chatbots 12th Symposium on Usable Privacy and Security, Denver, Colorado, US.
- Lewis, J. R. (2001). Psychometric properties of the mean opinion scale. Proceedings of HCI International 2001: Usability Evaluation and Interface Design, 149-153.
- Munteanu, C., & Boldea, M. (2000). MDWOZ: A Wizard of Oz Environment for Dialog Systems Development International Conference on Language Resources and Evaluation (LREC’00), Athens, Greece.
- Peras, D. (2018). Chatbot evaluation metrics. Economic and Social Development: Book of Proceedings, 89-97
- Radziwill, N. M., & Benton, M. C. (2017). Evaluating quality of chatbots and intelligent conversational agents. arXiv preprint arXiv:1704.04579. · arXiv
- Revelle W (2011) An overview of the psych package. Department of Psychology Northwestern University. Accessed on March 3(2012):1–25
- Saenz, J., Burgess, W., Gustitis, E., Mena, A., & Sasangohar, F. (2017). The usability analysis of chatbot technologies for internal personnel communications Industrial and Systems Engineering Conference Pittsburgh, Pennsylvania, US.
- Tay L, Jebb A (2017) Scale Creation. In: Rogelberg SG (ed) The SAGE encyclopedia of industrial and organizational psychology, 2nd edn. SAGE Publications, Inc.
- Wilson HJ, Daugherty P, Bianzino N (2017) The jobs that artificial intelligence will create. MIT Sloan Management Review 58(4):14
Cited by
Artificial intelligence applications in Latin American higher education: a systematic review
Salas-Pilco, S. Z., & Yang, Y. (2022) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-022-00326-w · Cited by 330
“These applications are being integrated in social networks, websites, and a variety of platforms (Borsci et al., 2021).”
BackgroundAl Naqbi, H., Bahroun, Z., & Ahmed, V. (2024) · Sustainability · doi:10.3390/su16031166 · Cited by 302
Nicolescu, L., & Tudorache, M. T. (2022) · Electronics · doi:10.3390/electronics11101579 · Cited by 280
Kim, J., & Im, I. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107512 · Cited by 220
Shahzad, M. F., Xu, S., An, X., & Javed, I. (2024) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2024.103867 · Cited by 202
Metadata
- Authors
- Borsci, S., Malizia, A., Schmettow, M., van der Velde, F., Tariverdiyeva, G., Balaji, D., & Chamberlain, A.
- Affiliations
- Imperial College London (GB)
- University of Pisa (IT)
- University of Twente (NL)
- University of Amsterdam (NL)
- University of Nottingham (GB)
- Corresponding author
- Chamberlain, Alan
- Venue
- Personal and Ubiquitous Computing
- Published
- 2021-07-21
- Volume
- 26
- Issue
- 1
- Pages
- 95-119
- ISSN
- 1617-4909 (ISSN-L)
- 1617-4917
- OpenAlex
- W3187911027
- Cited by
- 266
- References
- 68
- Citations by year
- 2026: 87 · 2025: 78 · 2024: 52 · 2023: 39 · 2022: 9 · 2021: 1
- FWCI
- 8.89
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Personal and Ubiquitous Computing (publishedVersion, cc-by)
- Repository@Nottingham (University of Nottingham) (publishedVersion, other-oa)
- University of Twente Research Information (publishedVersion, cc-by)
- Repository@Nottingham (University of Nottingham) (submittedVersion, cc-by)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion, other-oa)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- University of Twente Research Information (submittedVersion, cc-by)
- Funders
- Engineering and Physical Sciences Research Council (EP/T51729X/1)
- Topics
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- chatbots, user satisfaction, accessibility, chatbot evaluation, psychometric properties
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.