“Dave...I can assure you ...that it’s going to be all right ...” A Definition, Case for, and Survey of Algorithmic Assurances in Human-Autonomy Trust Relationships
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
People who design, use, and are affected by autonomous artificially intelligent agents want to be able to trust such agents—that is, to know that these agents will perform correctly, to understand the reasoning behind their actions, and to know how to use them appropriately. Many techniques have been devised to assess and influence human trust in artificially intelligent agents. However, these approaches are typically ad hoc and have not been formally related to each other or to formal trust models. This article presents a survey of algorithmic assurances , i.e., programmed components of agent operation that are expressly designed to calibrate user trust in artificially intelligent agents. Algorithmic assurances are first formally defined and classified from the perspective of formally modeled human-artificially intelligent agent trust relationships. Building on these definitions, a synthesis of research across communities such as machine learning, human-computer interaction, robotics, e-commerce, and others reveals that assurance algorithms naturally fall along a spectrum in terms of their impact on an agent’s core functionality, with seven notable classes ranging from integral assurances (which impact an agent’s core functionality) to supplemental assurances (which have no direct effect on agent performance). Common approaches within each of these classes are identified and discussed; benefits and drawbacks of different approaches are also investigated.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Humans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 3×
“Parasuraman and Riley [106] were interested in understanding the use of automation by humans, and defined terms to describe that use.”
BackgroundLewis, J. D., & Weigert, A. (1985) · Social Forces · doi:10.2307/2578601 · Cited by 4,163 · cited 1×
“elated Behaviors Something that is well accepted among researchers of all disciplines is that trust ultimately leads to some kind of behavior or action; this idea was highlighted by Lewis and Weigert [77]. McKnight and Chervany [88] call these ‘trust-related behaviors‘ (TRBs), which is the term that will be used in th…”
BackgroundTrust And Distrust: New Relationships and Realities
Lewicki, R. J., McAllister, D. J., & Bies, R. J. (1998) · The Academy of Management Review · doi:10.5465/amr.1998.926620 · Cited by 3,091
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088
Baier, A. (1986) · Ethics · doi:10.1086/292745 · Cited by 2,567
What trust means in e-commerce customer relationships: an interdisciplinary conceptual typology
McKnight, D. H., & Chervany, N. L. (2001) · International Journal of Electronic Commerce · doi:10.1080/10864415.2001.11044235 · Cited by 2,282 · cited 7×
“For instance, we consider the full trust model presented by McKnight and Chervany [73], whereas Lillard et al.”
BackgroundMethodInfluentialCan we open the black box of AI?
Castelvecchi, D. (2016) · Nature · doi:10.1038/538020a · Cited by 1,773 · cited 1×
“With the rapid advancement of intelligent technology to do tasks that were previously assumed to be too complicated for machines, there is now much discussion in public [24, 97, 110], business [6, 54, 101], and academic seings [12, 30] on how humans can trust said technology – although, the connection to trust is not…”
BackgroundDataset Shift in Machine Learning
(2008) · doi:10.7551/mitpress/9780262170055.001.0001 · Cited by 1,553
Concrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314
Initial trust formation in new organizational relationships
McKnight, D. H., Cummings, L. L., & Chervany, N. L. (1998) · The Academy of Management Review · doi:10.2307/259290 · Cited by 1,296
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 26×
“Rather, it is a multidimensional phenomenon whose ‘relative magnitudes and directions’ must be observed through changes in TRBs, or qualitative self-reports gathered via surveys [102].”
BackgroundInfluentialInformation-based objective functions for active data selection
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.4.590 · Cited by 1,221
On-line trust: concepts, evolving themes, a model
Corritore, C. L., Kracher, B., & Wiedenbeck, S. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00041-7 · Cited by 1,200 · cited 2×
“king a diagram that indicates that these interventions could a‡ect the ‘Trusting Beliefs’, ‘Trusting Intentions’, and ‘Trust-Related Behaviors’ (see Figure5) of an online human user. Corritore et al. [22] refer to assurances as ‘trust cues’ that can in…uence how online users trust vendors in an e-commerce se−ing. Lee…”
BackgroundMethodTrust in a specific technology: An investigation of its components and measures
Mcknight, D. H., Carter, M., Thatcher, J. B., & Clay, P. F. (2011) · ACM Transactions on Management Information Systems · doi:10.1145/1985347.1985353 · Cited by 1,185 · cited 2×
“Assessing self-reported changes in trust involves asking users to answer questions, such as ‘how trustworthy do you feel the system is?’; or ‘to what extent do you nd the system to be interpretable?’, either while using a system or aerwards [66, 89, 100, 117, 139].”
BackgroundMUIR, B. M., & MORAY, N. (1996) · Ergonomics · doi:10.1080/00140139608964474 · Cited by 993
MUIR, B. M. (1994) · Ergonomics · doi:10.1080/00140139408964957 · Cited by 873 · cited 4×
“Muir [98, 99] considered similar questions and concluded that making systems more ‘observable’ is one way to improve the trust between a human operator and an autonomous control system.”
BackgroundInfluentialThe Building Blocks of Interpretability
Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., & Mordvintsev, A. (2018) · Distill · doi:10.23915/distill.00010 · Cited by 591 · cited 1×
“Specifically, Olah et al. [104] investigate how predictions of NNs can be explained through visualizing how different parts of the network respond Manuscript submitted to ACM to certain images. propose combining several different approaches to get a holistic view of the NN behavior.”
BackgroundSalem, M., Lakatos, G., Amirabdollahian, F., & Dautenhahn, K. (2015) · Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/2696454.2696497 · Cited by 494 · cited 28×
“Rather, it is a multidimensional phenomenon whose ‘relative magnitudes and directions’ must be observed through changes in TRBs, or qualitative self-reports gathered via surveys [102].”
BackgroundInfluentialHuysmans, J., Dejaeger, K., Mues, C., Vanthienen, J., & Baesens, B. (2011) · Decision Support Systems · doi:10.1016/j.dss.2010.12.003 · Cited by 425 · cited 2×
“[49] compared decision trees, decision tables, propositional if-then rules, and oblique rule sets to understand which set of methods is ‘most interpretable’.”
BackgroundA Big Data Guide to Understanding Climate Change: The Case for Theory-Guided Data Science
Faghmous, J. H., & Kumar, V. (2014) · Big Data · doi:10.1089/big.2014.0026 · Cited by 223 · cited 2×
“for scientic data analysis [37], public policy and medicine [64, 138] and cognitive assistance [52].”
BackgroundMethodEffects of changing reliability on trust of robot systems
Desai, M., Medvedev, M., Vázquez, M., McSheehy, S., Gadea-Omelchenko, S., Bruggeman, C., Steinfeld, A., & Yanco, H. (2012) · Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction · doi:10.1145/2157689.2157702 · Cited by 188
Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S., & Dragan, A. (2017) · arXiv · doi:10.48550/arxiv.1711.02827 · Cited by 125
Kwon, M., Huang, S. H., & Dragan, A. D. (2018) · Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/3171221.3171276 · Cited by 123
(2018) · doi:10.1201/9781315137957-2 · Cited by 108
Unreliable Automated Attention Cueing for Air-Ground Targeting and Traffic Maneuvering
Wickens, C. D., Conejo, R., & Gempler, K. (1999) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193129904300105 · Cited by 29 · cited 2×
“There are two known approaches to measuring the effects of assurances: gathering self-reported changes [67, 92, 102, 119, 144] and measuring changes in TRBs [8, 32, 39, 119, 145].”
Background
Cited by
Explainable artificial intelligence: A survey
Dosilovic, F. K., Brcic, M., & Hlupic, N. (2018) · 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO) · doi:10.23919/mipro.2018.8400040 · Cited by 1,147
Towards a Theory of Longitudinal Trust Calibration in Human–Robot Teams
de Visser, E. J., Peeters, M. M. M., Jung, M. F., Kohn, S., Shaw, T. H., Pak, R., & Neerincx, M. A. (2020) · International Journal of Social Robotics · doi:10.1007/s12369-019-00596-x · Cited by 447
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. (2021) · arXiv · doi:10.48550/arxiv.2112.11561 · Cited by 336
Atakishiyev, S., Salameh, M., Yao, H., & Goebel, R. (2024) · IEEE Access · doi:10.1109/access.2024.3431437 · Cited by 323
Górriz, J. M., Ramírez, J., Ortíz, A., Martínez-Murcia, F. J., Segovia, F., Suckling, J., Leming, M., Zhang, Y.-D., Álvarez-Sánchez, J. R., Bologna, G., et al. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.05.078 · Cited by 319
Metadata
- Authors
- Israelsen, B. W., & Ahmed, N. R.
- Venue
- ACM Computing Surveys
- Published
- 2019-01-28
- Volume
- 51
- Issue
- 6
- Pages
- 1-37
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2740720719
- Cited by
- 74
- References
- 27
- FWCI
- 9.98
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.