The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
In bot we trust: A new methodology of chatbot performance measures
Abstract
Abstract Chatbots are used frequently in business to facilitate various processes, particularly those related to customer service and personalization. In this article, we propose novel methods of tracking human-chatbot interactions and measuring chatbot performance that take into consideration ethical concerns, particularly trust. Our proposed methodology links neuroscientific methods, text mining, and machine learning. We argue that trust is the focal point of successful human-chatbot interaction and assess how trust as a relevant category is being redefined with the advent of deep learning supported chatbots. We propose a novel method of analyzing the content of messages produced in human-chatbot interactions, using the Condor Tribefinder system we developed for text mining that is based on a machine learning classification engine. Our results will help build better social bots for interaction in business or commercial environments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Does the chimpanzee have a theory of mind?
Premack, D., & Woodruff, G. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00076512 · Cited by 8,743
Fiske, S. T., Cuddy, A. J. C., Glick, P., & Xu, J. (2002) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.82.6.878 · Cited by 7,364
Wimmer, H. (1983) · Cognition · doi:10.1016/0010-0277(83)90004-5 · Cited by 6,305
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
The Uncanny Valley [From the Field]
Mori, M., MacDorman, K., & Kageki, N. (2012) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2012.2192811 · Cited by 3,542
Kaplan, A., & Haenlein, M. (2019) · Business Horizons · doi:10.1016/j.bushor.2018.08.004 · Cited by 3,140
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,738
The mind in the machine: Anthropomorphism increases trust in an autonomous vehicle
Waytz, A., Heafner, J., & Epley, N. (2014) · Journal of Experimental Social Psychology · doi:10.1016/j.jesp.2014.01.005 · Cited by 1,439
Feeling and Believing: The Influence of Emotion on Trust.
Dunn, J. R., & Schweitzer, M. E. (2005) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.88.5.736 · Cited by 1,301
Tribal marketing: The tribalisation of society and its impact on the conduct of marketing
Cova, B., & Cova, V. (2002) · European Journal of Marketing · doi:10.1108/03090560210423023 · Cited by 888
Computer Power and Human Reason: From Judgment to Calculation
Agassi, J., & Wiezenbaum, J. (1976) · Technology and Culture · doi:10.2307/3103715 · Cited by 877
Computer Power and Human Reason: From Judgement to Calculation.
Crowther, J. F., & Weizenbaum, J. (1977) · Contemporary Sociology · doi:10.2307/2062802 · Cited by 722
In the shades of the uncanny valley: An experimental study of human–chatbot interaction
Ciechanowski, L., Przegalinska, A., Magnuski, M., & Gloor, P. (2019) · Future Generation Computer Systems · doi:10.1016/j.future.2018.01.055 · Cited by 651
Self-citationChatbots: Are they Really Useful?
Abu Shawar, B., & Atwell, E. (2007) · Journal for Language Technology and Computational Linguistics · doi:10.21248/jlcl.22.2007.88 · Cited by 616
Baron, J. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511571275 · Cited by 507
Relational agents: a model and implementation of building user trust
Bickmore, T., & Cassell, J. (2001) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365304 · Cited by 479
Mori, M. (2020) · The Monster Theory Reader · doi:10.5749/j.ctvtv937f.7 · Cited by 433
Evans, J. S. B. T., & Baron, J. (1987) · British Journal of Educational Studies · doi:10.2307/3120822 · Cited by 370
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
(2002) · doi:10.1145/503376 · Cited by 306
López, G., Quesada, L., & Guerrero, L. A. (2018) · Advances in Intelligent Systems and Computing · doi:10.1007/978-3-319-60366-7_23 · Cited by 277
External manifestations of trustworthiness in the interface
Cassell, J., & Bickmore, T. (2000) · Communications of the ACM · doi:10.1145/355112.355123 · Cited by 255
Chung, H., Iorga, M., Voas, J., & Lee, S. (2017) · Computer · doi:10.1109/mc.2017.3571053 · Cited by 230
Asking ‘Why’ in AI: Explainability of intelligent systems – perspectives and challenges
Preece, A. D. (2018) · Intelligent systems in accounting, finance and management/Intelligent systems in accounting, finance & management · doi:10.1002/isaf.1422 · Cited by 212
Using corpora in machine-learning chatbot systems
Shawar, B. A., & Atwell, E. S. (2005) · International Journal of Corpus Linguistics · doi:10.1075/ijcl.10.4.06sha · Cited by 189
Kacprzyk, J., & Zadrozny, S. (2010) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2010.2040480 · Cited by 184
Evaluating Quality of Chatbots and Intelligent Conversational Agents
Radziwill, N., & Benton, M. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.04579 · Cited by 181
Persuasiveness of expert systems
Dijkstra, J. J., Liebrand, W. B. G., & Timminga, E. (1998) · Behaviour & Information Technology · doi:10.1080/014492998119526 · Cited by 161
Different measurements metrics to evaluate a chatbot system
Shawar, B. A., & Atwell, E. (2007) · doi:10.3115/1556328.1556341 · Cited by 151
The World is not Enough: Trust in Cognitive Agents
de Visser, E. J., Krueger, F. W., McKnight, P. E., Scheid, S., Smith, M. A. B., Chalk, S., & Parasuraman, R. (2012) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1071181312561062 · Cited by 136
Can machines talk? Comparison of Eliza with modern dialogue systems
Shah, H., Warwick, K., Vallverdú, J., & Wu, D. (2016) · Computers in Human Behavior · doi:10.1016/j.chb.2016.01.004 · Cited by 133
Hernández‐Orallo, J. (2000) · Journal of Logic Language and Information · doi:10.1023/a:1008367325700 · Cited by 131
Surf tribal behaviour: a sports marketing application
Moutinho, L., Dionísio, J., & Leal, M. D. C. (2007) · Marketing Intelligence & Planning · doi:10.1108/02634500710834160 · Cited by 112
‘Realness’ in Chatbots: Establishing Quantifiable Criteria
Morrissey, K., & Kirakowski, J. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39330-3_10 · Cited by 110
Natural language dialogue for personalized interaction
Zadrozny, W., Budzikowska, M., Chai, J. Y., Kambhatla, N., Levesque, S., & Nicolov, N. (2000) · Communications of the ACM · doi:10.1145/345124.345164 · Cited by 75
The Status and Future of the Turing Test
Moor, J. H. (2001) · Minds and Machines · doi:10.1023/a:1011218925467 · Cited by 74
Trust in Technology: A Distinctive and a Problematic Relation
Taddeo, M. (2010) · Knowledge, technology & policy/Knowledge in society/Knowledge, technology, & policy · doi:10.1007/s12130-010-9113-9 · Cited by 56
Polish Version of the Negative Attitude Toward Robots Scale (NARS-PL)
Pochwatko, G., Giger, J., Różańska-Walczuk, M., Świdrak, J., Kukiełka, K., Możaryn, J., & Piçarra, N. J. G. (2015) · Journal of Automation Mobile Robotics & Intelligent Systems · doi:10.14313/jamris_3-2015/25 · Cited by 50
The Future of work: The meaning and value of work in EuropePreprint
Méda, D. (2016) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 47
Seeger, A.-M., & Heinzl, A. (2017) · Lecture notes in information systems and organisation · doi:10.1007/978-3-319-67431-5_15 · Cited by 46
From Chatbots to Dialog Systems
Klüwer, T. (2011) · IGI Global eBooks · doi:10.4018/978-1-60960-617-6.ch001 · Cited by 41
French, R. M. (2012) · Communications of the ACM · doi:10.1145/2380656.2380674 · Cited by 35
The World is not Enough: Trust in Cognitive Agents: (572172013-055)
de Visser, E. J., Krueger, F., McKnight, P., Scheid, S., Smith, M., Chalk, S., & Parasuraman, R. (2012) · PsycEXTRA Dataset · doi:10.1037/e572172013-055 · Cited by 34
Khan, R., & Das, A. (2017) · Apress eBooks · doi:10.1007/978-1-4842-3111-1_1 · Cited by 33
Trust in chatbots for customer service – findings from a questionnaire study
Nordheim, C. B. (2018) · NORA - Norwegian Open Research Archives · Cited by 30
The Necessity of New Paradigms in Measuring Human-Chatbot Interaction
Ciechanowski, L., Przegalińska, A., & Wegner, K. (2017) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-60747-4_19 · Cited by 26
Self-citationAdvances in Human Factors and Systems Interaction
(2018) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-94334-3 · Cited by 20
Collaborative Innovation Networks
(2018) · Studies on entrepreneurship, structural change and industrial dynamics · doi:10.1007/978-3-319-74295-3 · Cited by 20
Machine learning and deep analytics for biocomputing: call for better explainability
Petković, D., Kobzik, L., & Ré, C. (2017) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789813235533_0058 · Cited by 13
GalaxyScope: Finding the “Truth of Tribes” on Social Media
de Oliveira, J. M., & Gloor, P. A. (2018) · Studies on entrepreneurship, structural change and industrial dynamics · doi:10.1007/978-3-319-74295-3_13 · Cited by 11
Self-citationDeveloping a Corporate Chatbot for a Customer Engagement Program: A Roadmap
Castro, F. P. D. S., Tedesco, P. C. D. A. R., Alves, H. D., Quintino, J. P., Steffen, J., de Oliveira, F. S., Soares, R. A., Santos, A. L. M., & da Silva, F. Q. B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-95930-6_37 · Cited by 10
Weng, F., Wang, Y., Tur, G., & Hu, J. (2007) · Cited by 10
Stojnić, A. (2015) · Performance Research · doi:10.1080/13528165.2015.1026733 · Cited by 9
A Warmer Welcome : Application of a Chatbot as a Facilitator for New Hires Onboarding
Asher, N. (2017) · DiVA (Linnaeus University) · Cited by 8
Rabinowitz, N. C., Perbet, F., Song, H. F., Zhang, C., Eslami, S. M. A., & Botvinick, M. (2018) · arXiv · doi:10.48550/arxiv.1802.07740
Cited by
Ashfaq, M., Yun, J., Yu, S., & Loureiro, S. M. C. (2020) · Telematics and Informatics · doi:10.1016/j.tele.2020.101473 · Cited by 987
Rediscovering the use of chatbots in education: A systematic literature review
Pérez, J. Q., Daradoumis, T., & Puig, J. M. M. (2020) · Computer Applications in Engineering Education · doi:10.1002/cae.22326 · Cited by 483
A Literature Survey of Recent Advances in Chatbots
Caldarini, G., Jaf, S., & McGarry, K. (2022) · Information · doi:10.3390/info13010041 · Cited by 470
Chen, J.-S., Le, T.-T.-Y., & Florence, D. (2021) · International Journal of Retail & Distribution Management · doi:10.1108/ijrdm-08-2020-0312 · Cited by 431
Youn, S., & Jin, S. V. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106721 · Cited by 317
Metadata
- Authors
- Przegalinska, A., Ciechanowski, L., Stroz, A., Gloor, P., & Mazurek, G.
- Venue
- Business Horizons
- Publisher
- Elsevier BV
- Published
- 2019-11
- Volume
- 62
- Issue
- 6
- Pages
- 785-797
- ISSN
- 0007-6813 (ISSN-L)
- OpenAlex
- W2974124990
- Cited by
- 344
- References
- 54
- FWCI
- 13.24
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.