The Chatbot Usability Scale: the Design and Pilot of a Usability Scale for Interaction with AI-Based Conversational Agents
Open accessThe DOI matches its Crossref record; 67 of the 68 references it cites were found in a registry or an index.
Abstract
Standardised tools to assess a user’s satisfaction with the experience of using chatbots and conversational agents are currently unavailable. This work describes four studies, including a systematic literature review, with an overall sample of 141 participants in the survey (experts and novices), focus group sessions and testing of chatbots to (i) define attributes to assess the quality of interaction with chatbots and (ii) the designing and piloting a new scale to measure satisfaction after the experience with chatbots. Two instruments were developed: (i) A diagnostic tool in the form of a checklist (BOT-Check). This tool is a development of previous works which can be used reliably to check the quality of a chatbots experience in line with commonplace principles. (ii) A 15-item questionnaire (BOT Usability Scale, BUS-15) with estimated reliability between .76 and .87 distributed in five factors. BUS-15 strongly correlates with UMUX-LITE by enabling designers to consider a broader range of aspects usually not considered in satisfaction tools for non-conversational agents, e.g. conversational efficiency and accessibility, quality of the chatbot’s functionality and so on. Despite the convincing psychometric properties, BUS-15 requires further testing and validation. Designers can use it as a tool to assess products, thus building independent databases for future evaluation of its reliability, validity and sensitivity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989
Usability Evaluation In Industry
(1996) · CRC Press eBooks · doi:10.1201/9781498710411 · Cited by 4,354
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · cited 2×
“The agreement among the participants was equal to 0.780, which is acceptably higher than the minimum level of .667 for the Krippendorff ’ s Alpha [35].”
MethodResultInfluentialFitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017) · JMIR Mental Health · doi:10.2196/mental.7785 · Cited by 2,730
The measurement of user information satisfaction
Ives, B., Olson, M. H., & Baroudi, J. J. (1983) · Communications of the ACM · doi:10.1145/358413.358430 · Cited by 1,820
Service loyalty: The effects of service quality and the mediating role of customer satisfaction
Caruana, A. (2002) · European Journal of Marketing · doi:10.1108/03090560210430818 · Cited by 1,528
Trust in a specific technology: An investigation of its components and measures
Mcknight, D. H., Carter, M., Thatcher, J. B., & Clay, P. F. (2011) · ACM Transactions on Management Information Systems · doi:10.1145/1985347.1985353 · Cited by 1,185
Go, E., & Sundar, S. S. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.01.020 · Cited by 1,089
Research works printed but not linked: 13
- Charlton JI (2000) Nothing about us without us: Disability oppression and empowerment. Univ of California Press
- DeVellis, R. F. (2016). Scale development: Theory and applications (Vol. 26). Sage publications.
- Dillon A (2001) Beyond Usability: Process, Outcome and Affect in human computer interactions. Canadian Journal of Information and Library Science 26(4)
- Feine, J., Morana, S., & Gnewuch, U. (2019). Measuring service encounter satisfaction with customer service chatbots using sentiment analysis 14th International Conference on Wirtschaftsinformatik (WI2019), Siegen, Germany. https://aisel.aisnet.org/wi2019/
- Harkous, H., Fawaz, K., Shin, K. G., & Aberer, K. (2016). Pribots: Conversational privacy with chatbots 12th Symposium on Usable Privacy and Security, Denver, Colorado, US.
- Lewis, J. R. (2001). Psychometric properties of the mean opinion scale. Proceedings of HCI International 2001: Usability Evaluation and Interface Design, 149-153.
- Munteanu, C., & Boldea, M. (2000). MDWOZ: A Wizard of Oz Environment for Dialog Systems Development International Conference on Language Resources and Evaluation (LREC’00), Athens, Greece.
- Peras, D. (2018). Chatbot evaluation metrics. Economic and Social Development: Book of Proceedings, 89-97
- Radziwill, N. M., & Benton, M. C. (2017). Evaluating quality of chatbots and intelligent conversational agents. arXiv preprint arXiv:1704.04579. · arXiv
- Revelle W (2011) An overview of the psych package. Department of Psychology Northwestern University. Accessed on March 3(2012):1–25
- Saenz, J., Burgess, W., Gustitis, E., Mena, A., & Sasangohar, F. (2017). The usability analysis of chatbot technologies for internal personnel communications Industrial and Systems Engineering Conference Pittsburgh, Pennsylvania, US.
- Tay L, Jebb A (2017) Scale Creation. In: Rogelberg SG (ed) The SAGE encyclopedia of industrial and organizational psychology, 2nd edn. SAGE Publications, Inc.
- Wilson HJ, Daugherty P, Bianzino N (2017) The jobs that artificial intelligence will create. MIT Sloan Management Review 58(4):14
Cited by
Items 301–361 of 361
Zhou, H., Brown, J. A., Lush, B., & Moroney, T. L. (2026) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000001610
Assessing Conversational AI Usability: A System Usability Scale Evaluation of Dola AI
Kurniasari, A., & Velisya, N. A. (2026) · Jurnal Ilmiah Teknik · doi:10.56127/juit.v5i2.1324
Katake, K. J. (2026) · Journal of Intelligent Decision Making and Information Science · doi:10.59543/jidmis.v3.1968
Zheng, Y., Debs, L., & Besiktepe, D. (2026) · ASCE Construction Research Congress (CRC) · doi:10.1061/9780784486993.023
Kadri, M., Alaoui, S., Hnini, A., & Chlioui, I. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-28573-7_49
Exploring AI-Driven Scaffolding for Critical Questioning in Argument Evaluation
Belete, E. A. (2026) · Digital Scholarship - UNLV (University of Nevada Reno)
Fostering Occupational Cognition in Children: An LLM-Powered Multi-Agent Role-Playing Approach
Zhou, Y., Xi, H., Ma, Z., & Yao, C. (2025) · International Symposium on Computational Intelligence and Design · doi:10.1109/iscid68789.2025.00064
Israr, A., Ahrar, M. U., khan, F., Aziz, A., & Ahmed, F. (2026) · TAMSAAL · doi:10.59075/tamsaal.v4i8.139
Predictive Analysis of User Experience in Conversational Design Applications
Venugopal, K. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-0512-5.ch004
Development and psychometric evaluation of the AI emotional engagement scale
Xu, J., Mao, M., Yao, Y., Wen, L., & Dai, Q. (2026) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2026.1921527
WAR-MATE: A Tactical Conversational Decision Framework for High-Pressure Systems
Tharun, P., Patil, K. P. S., & Anu, D. (2026) · International Conference on Inventive Systems and Control (ICISC) · doi:10.1109/icisc69558.2026.11681537
Shen, A., Xu, Z., Ma, Z., Yin, M., Lian, B., Chen, Y., Fan, H., & Chai, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7478078
Formative and summative use of automatic conversational assessment in higher education
Shingjergji, K., Bauer, J., Härmä, A., Verstegen, D. M. L., Arts, J., & Brinke, D. J.-T. (2026) · Assessment in Education Principles Policy and Practice · doi:10.1080/0969594x.2026.2736624
Kuštelega, M., & Mekovec, R. (2026) · Society and Economy · doi:10.14267/1588970x.2026.027
Latif, M. S., Li, G., Wang, J., & Shahzad, M. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2731487
Gao, L., Fischer, F., Hofmann, R., Kılıçkesmez, G., & Chernikova, O. (2026) · Open access LMU (Ludwid Maxmilian's Universitat Munchen) · doi:10.5282/ubm/epub.138007
Leppla, L. C., Spanier, M., & Seibert, J. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1952302
Editor, P. J., Gupta, T., Khandelwal, R., Sharma, R., & Sharma, N. (2024) · PRATIBODH
Weighting Customer Perceptions of AI Chatbots in E-Commerce: An AHP-Based Evaluation Framework
Nikith, N., & Prasad, A. V. S. (2026) · International Journal of Innovative Science and Research Technology (IJISRT) · doi:10.38124/ijisrt/26sep1276
Chuaichum, C., Ngudgratoke, S., Nakorn, N. N., & Saenpich, S. (2026) · Journal of Social Science and Cultural
SISTEM PELAYANAN NIKAH BERBASIS WEB DENGAN ASISTEN VIRTUAL DI KECAMATAN SIRIMAU
Siami, M. I. (2024) · Jurnal Ilmu Komputer STIKOM Ambon
Zhang, F., Chen, Y., Liu, M., Xu, J., & Xu, J. (2026) · Scientific Reports · doi:10.1038/s41598-026-72496-w
Thanajaro, N. (2026) · KKBS Journal of Business Administration and Accountancy
Mir, R. N., & Hussain, A. (2026) · Journal of Nursing and Allied Health · doi:10.37939/jnah.v4i03.269
Jent, J. F., Mannweiler, M. D., Malave, K. Y., Mirza, A., Ogihara, M., Alonso, B., Juan, J. S., Davis, E., Sobalvarro, C., & Garcia, D. (2026) · JMIR Formative Research · doi:10.2196/105131
Chen, Y., Li, Y., Xiong, Y., Ji, S., Ding, X. X., Deng, Y., & Fan, Y. (2026) · Digital Health · doi:10.1177/20552076261495459
Reducing Cognitive Load to Strengthen Customer Engagement: The Strategic Role of Digital Tools
Mossaei, N., & Tofighi, M. (2026) · DigitalCommons - Kennesaw State University (Kennesaw State University)
Gomes, L. M. F., Bernardi, G., & Cordenonsi, A. Z. (2026) · Navus - Revista de Gestão e Tecnologia · doi:10.22279/navus.v18.2426
Perceived Response Relevance, Transparency, and Interaction Ease on CLEO Chatbot Trust in Jakarta
Sarwar, A. A. M. (2026) · Digitus Journal of Computer Science Applications · doi:10.61978/digitus.v4i2.2038
Chau, S. L., Janet Yuen Ha, W. O. N. G., Fong, D. Y. T., Ho, J. W. K., Tzu Tsun, L. U. K., Wong, C. K. H., Ho, S. Y., Lam, T. H., & Ping, W. M. (2026) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2026.113391
BronchoSAS: A State-Aware Agent with Skills for Phantom-Based Bronchoscopy Training
Tian, C., Yu, H., Oliveira, B., Jiang, H., Konge, L., Cold, K. M., & Cheng, Z. (2026) · WRC Symposium on Advanced Robotics and Automation (WRC SARA) · doi:10.1109/wrcsara71458.2026.11732584
Annamalai, S., & Tan, J. W. (2026) · International Journal of Innovation and Industrial Revolution · doi:10.35631/ijirev.826040
Human-Centered Agentic AI Applications: Characteristics and Design Patterns
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805748
(2026) · ACM eBooks · doi:10.1145/3805739.3805741
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805753
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805752
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805747
Human-Centered Agentic AI Applications in Healthcare
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805750
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805744
윤희정, Ahn, G., & Kim, M. (2026) · Asia-pacific Journal of Convergent Research Interchange · doi:10.47116/apjcri.2026.10.81
(2026) · ACM eBooks · doi:10.1145/3805739.3805757
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805746
(2026) · ACM eBooks · doi:10.1145/3805739.3805740
Agentic AI Solutions for Human Resource Management
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805756
Agentic AI Solutions for Education
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805754
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805743
(2026) · ACM eBooks · doi:10.1145/3805739.3805758
Human-Centered Agentic AI Applications in Human Resource Management
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805751
Human-Centered Agentic AI Applications in Education
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805749
Agentic AI Solutions for Healthcare
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805755
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805745
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739
Zhou, M. X., Lu, J., Huahai, Y., & Chen, W. (2026) · ACM eBooks · doi:10.1145/3805739.3805742
Lu, T., & Song, J.-S. (2026) · Scientific Reports · doi:10.1038/s41598-026-77412-w
Ownby, R. L., Caballero, J., & Thomas-Purcell, K. B. (2026) · medRxiv · doi:10.64898/2026.11.05.26367380
Jomaa, S. (2026) · Digital Commons (Liberty University)
Modality Matters: How Interaction Modality Shapes Older Adults’ Perceptions of Conversational Agents
Nanayakkara, S., Perera, K. J. P. G., & Huff, E. W. (2026) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3841580.3841586
CheOmar, A. (2026) · Prosiding Seminar Nasional Teknologi Komputer dan Sains
S, F. L. (2026) · Journal of Sustainable Technology in Agriculture · doi:10.65287/josta.202610.de78
Juniad, S., Malik, A. N., Liaqat, M., Ameer, S. M., & Khan, S. (2026) · Journal of Innovative Research in Management Sciences · doi:10.62270/jirms.v7i3.149
Analysis of chat-communication instruments utilization in Russian libraries
Smirnov, Y. V. (2026) · Scientific and Technical Libraries · doi:10.33186/1027-3689-2026-10-180-200
Metadata
- Authors
- Borsci, S., Malizia, A., Schmettow, M., van der Velde, F., Tariverdiyeva, G., Balaji, D., & Chamberlain, A.
- Affiliations
- Imperial College London (GB)
- University of Pisa (IT)
- University of Twente (NL)
- University of Amsterdam (NL)
- University of Nottingham (GB)
- Corresponding author
- Chamberlain, Alan
- Venue
- Personal and Ubiquitous Computing
- Published
- 2021-07-21
- Volume
- 26
- Issue
- 1
- Pages
- 95-119
- ISSN
- 1617-4909 (ISSN-L)
- 1617-4917
- OpenAlex
- W3187911027
- Cited by
- 266
- References
- 68
- Citations by year
- 2026: 87 · 2025: 78 · 2024: 52 · 2023: 39 · 2022: 9 · 2021: 1
- FWCI
- 8.89
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Personal and Ubiquitous Computing (publishedVersion, cc-by)
- Repository@Nottingham (University of Nottingham) (publishedVersion, other-oa)
- University of Twente Research Information (publishedVersion, cc-by)
- Repository@Nottingham (University of Nottingham) (submittedVersion, cc-by)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion, other-oa)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- University of Twente Research Information (submittedVersion, cc-by)
- Funders
- Engineering and Physical Sciences Research Council (EP/T51729X/1)
- Topics
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- chatbots, user satisfaction, accessibility, chatbot evaluation, psychometric properties
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.