Evaluating Quality of Chatbots and Intelligent Conversational Agents
Open accessThe DOI matches its OpenAlex record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
Chatbots are one class of intelligent, conversational software agents activated by natural language input (which can be in the form of text, voice, or both). They provide conversational output in response, and if commanded, can sometimes also execute tasks. Although chatbot technologies have existed since the 1960s and have influenced user interface development in games since the early 1980s, chatbots are now easier to train and implement. This is due to plentiful open source code, widely available development platforms, and implementation options via Software as a Service (SaaS). In addition to enhancing customer experiences and supporting learning, chatbots can also be used to engineer social harm - that is, to spread rumors and misinformation, or attack people for posting their thoughts and opinions online. This paper presents a literature review of quality issues and attributes as they relate to the contemporary issue of chatbot development and implementation. Finally, quality assessment approaches are reviewed, and a quality assessment method based on these attributes and the Analytic Hierarchy Process (AHP) is proposed and examined.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The evaluation tool of identified chatbots was developed using the framework developed by Radziwill and Benton [36] for evaluating chatbots, which is grounded in the computer science literature.”
From Chatbots to Support People With Dementia and Their Caregivers: Systematic Review of Functions and Quality (2021) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Ferrara, E., Varol, O., Davis, C., Menczer, F., & Flammini, A. (2016) · Communications of the ACM · doi:10.1145/2818717 · Cited by 1,973
Miner, A. S., Milstein, A., Schueller, S., Hegde, R., Mangurian, C., & Linos, E. (2016) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.0400 · Cited by 368
(2016) · doi:10.1145/2950290.2983989 · Cited by 153
New technologies and mixed-use convergence: How humans and algorithms are adapting to each other
Applin, S. A., & Fischer, M. D. (2015) · 2015 IEEE International Symposium on Technology and Society (ISTAS) · doi:10.1109/istas.2015.7439436 · Cited by 51
Cited by
Ashfaq, M., Yun, J., Yu, S., & Loureiro, S. M. C. (2020) · Telematics and Informatics · doi:10.1016/j.tele.2020.101473 · Cited by 987
An overview of the features of chatbots in mental health: A scoping review
Abd-alrazaq, A. A., Alajlani, M., Alalwan, A. A., Bewick, B. M., Gardner, P., & Househ, M. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.103978 · Cited by 605
“18 language processing and growing availability of large health databases and corpora [51, 52], developers should endeavour to build more artificial intelligence chatbots.”
BackgroundConversational Agents in Health Care: Scoping Review and Conceptual Analysis
Tudor Car, L., Dhinagaran, D. A., Kyaw, B. M., Kowatsch, T., Joty, S., Theng, Y.-L., & Atun, R. (2020) · Journal of Medical Internet Research · doi:10.2196/17158 · Cited by 553
Rediscovering the use of chatbots in education: A systematic literature review
Pérez, J. Q., Daradoumis, T., & Puig, J. M. M. (2020) · Computer Applications in Engineering Education · doi:10.1002/cae.22326 · Cited by 483
A Literature Survey of Recent Advances in Chatbots
Caldarini, G., Jaf, S., & McGarry, K. (2022) · Information · doi:10.3390/info13010041 · Cited by 470
“[24], for example, define chatbots as a subcategory of conversational agents that are not usually embodied.”
BackgroundIn bot we trust: A new methodology of chatbot performance measures
Przegalinska, A., Ciechanowski, L., Stroz, A., Gloor, P., & Mazurek, G. (2019) · Business Horizons · doi:10.1016/j.bushor.2019.08.005 · Cited by 344
Schachner, T., Keller, R., & v Wangenheim, F. (2020) · Journal of Medical Internet Research · doi:10.2196/20701 · Cited by 267
“This emulated human-machine communication becomes more complex and sophisticated, especially through advancements in machine learning with the application of neural networks [13-15].”
BackgroundHuman-like communication in conversational agents: a literature review and research agenda
Van Pinxteren, M. M. E., Pluymaekers, M., & Lemmink, J. G. A. M. (2020) · Journal of Service Management · doi:10.1108/josm-06-2019-0175 · Cited by 280
Nguyen, D. M., Chiu, Y.-T. H., & Le, H. D. (2021) · Sustainability · doi:10.3390/su13147625 · Cited by 269
Social companionship with artificial intelligence: Recent trends and future avenues
Chaturvedi, R., Verma, S., Das, R., & Dwivedi, Y. K. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.122634 · Cited by 244
The effect of social presence and chatbot errors on trust
Toader, D.-C., Boca, G., Toader, R., Măcelaru, M., Toader, C., Ighian, D., & Rădulescu, A. T. (2019) · Sustainability · doi:10.3390/su12010256 · Cited by 233
Optimizing Students’ Mental Health and Academic Performance: AI-Enhanced Life Crafting
Dekker, I., De Jong, E. M., Schippers, M. C., De Bruijn-Smolders, M., Alexiou, A., & Giesbers, B. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.01063 · Cited by 176
“These techniques have the form of ‘supervised learning,’ using large datasets to train the chatbot which answers are appropriate and which are not; ‘unsupervised learning’ using Markov-chain based models; and ‘hybrid intelligence’ which combines both (c.f., Radziwill and Benton, 2017).”
MethodRahman, M. S., Bag, S., Hossain, M. A., Abdel Fattah, F. A. M., Gani, M. O., & Rana, N. P. (2023) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2023.103273 · Cited by 170
Conversational Agents: Goals, Technologies, Vision and Challenges
Allouch, M., Azaria, A., & Azoulay, R. (2021) · Sensors · doi:10.3390/s21248448 · Cited by 168
“Radziwill and Benton [14] present a literature review of quality issues related to CA development and implementation, focusing on two topics: quality-attributes and quality-assessment approaches.”
BackgroundReadiness for voice assistants to support healthcare delivery during a health crisis and pandemic
Sezgin, E., Huang, Y., Ramtekkar, U., & Lin, S. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00332-0 · Cited by 155
“Both enable communications with end-users via natural language, achieved through rule-based dialog or machine learning 6 .”
BackgroundNatural Language Generation Using Deep Learning to Support MOOC Learners
Li, C., & Xing, W. (2021) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-020-00235-x · Cited by 133
“Much research has suggested conversational agents had the potential to support students in online learning (Pereira 2016; Pereira et al. 2019; Radziwill and Benton 2017; Shawar and Atwell 2007b).”
BackgroundRuggiano, N., Brown, E. L., Roberts, L., Framil Suarez, C. V., Luo, Y., Hao, Z., & Hristidis, V. (2021) · Journal of Medical Internet Research · doi:10.2196/25006 · Cited by 100
The effect of conversational agent skill on user behavior during deception
Schuetzler, R. M., Grimes, G. M., & Giboney, J. S. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.03.033 · Cited by 95
Chow, J. C. L., & Li, K. (2024) · JMIR Bioinformatics and Biotechnology · doi:10.2196/64406 · Cited by 75
AI in healthcare: A narrative review
Väänänen, A., Haataja, K., Vehviläinen-Julkunen, K., & Toivanen, P. (2021) · F1000Research · doi:10.12688/f1000research.26997.2 · Cited by 69
Self-Anamnesis with a Conversational User Interface: Concept and Usability Study
Hochreutener, S., Pöpel, A., May, R., & Denecke, K. (2018) · Methods of Information in Medicine · doi:10.1055/s-0038-1675822 · Cited by 47
Chasing the Chatbots: Directions for Interaction and Design Research
Piccolo, L. S. G., Mensio, M., & Alani, H. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-17705-8_14 · Cited by 28
Development of a practical training method for a healthcare artificial intelligence (AI) chatbot
Lin, J., Joseph, T., Parga-Belinkie, J. J., Mandel, A., Schumacher, R., Neumann, K., Scalise, L., Gaulton, J., Christ, L., Leitner, K., et al. (2021) · BMJ Innovations · doi:10.1136/bmjinnov-2020-000530 · Cited by 13
Predicting Interaction Quality Aspects Using Level-Based Scores for Conversational Agents
Reimann, M. M., Oertel, C., Kunneman, F. A., & Hindriks, K. V. (2023) · Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents · doi:10.1145/3570945.3607332 · Cited by 4
“Most metrics proposed in the literature for evaluating the quality of conversational agents focus on the general quality of a dialogue and response generation [2, 19, 21].”
BackgroundMemory Fabric for Conversational AI Agents: Enabling Shared and Persistent Memory Across Users
Tiwari, A., & Gupta, V. (2025) · TechRxiv · doi:10.36227/techrxiv.176523350.08289935/v1 · Cited by 2
Metadata
- Authors
- Radziwill, N., & Benton, M. C.
- Venue
- arXiv (Cornell University)
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1704.04579
- OpenAlex
- W2606636926
- Cited by
- 181
- References
- 4
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.