Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
The DOI matches its Crossref record; 74 of the 76 references it cites were found in a registry or an index, and 2 are not checked yet.
Abstract
Trust is a central component of the interaction between people and AI, in that 'incorrect' levels of trust may cause misuse, abuse or disuse of the technology. But what, precisely, is the nature of trust in AI? What are the prerequisites and goals of the cognitive mechanism of trust, and how can we promote them, or assess whether they are being satisfied in a given interaction? This work aims to answer these questions. We discuss a model of trust inspired by, but not identical to, interpersonal trust (i.e., trust between people) as defined by sociologists. This model rests on two key properties: the vulnerability of the user; and the ability to anticipate the impact of the AI model's decisions. We incorporate a formalization of 'contractual trust', such that trust between a user and an AI model is trust that some implicit or explicit contract will hold, and a formalization of 'trustworthiness' (that detaches from the notion of trustworthiness in sociology), and with it concepts of 'warranted' and 'unwarranted' trust. We present the possible causes of warranted trust as intrinsic reasoning and extrinsic behavior, and discuss how to design trustworthy AI, how to evaluate whether trust has manifested, and whether it is warranted. Finally, we elucidate the connection between trust and XAI using our formalization.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Uncertainty is a defining characteristic of trust (Mayer et al., 1995) and has been referred to as a necessary prerequisite of human trust in AI that has been lacking in current XAI studies (Jacovi et al., 2021).”
From Exploring the effects of human-centered AI explanations on trust and reliance (2023) · cites this paper as Jacovi et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“research found that people often ignore recommendations because they do not trust them”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The primary motivation for clinical XAI is fostering appropriate trust: enabling clinicians to calibrate their reliance on AI recommendations based on the quality of supporting reasoning [156,157].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [157]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, people find it difficult to correctly calibrate their trust in a decision aid [10,13,21,37].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“For example, [38] formalizes human-AI trust by considering and extending the concepts of interpersonal trust.”
From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [38]
9 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736 · cited 1×
“on basic definition of trust regards it as a directional transaction between two parties: if A believes that B will act in A’s best interest, and accepts vulnerability to B’s actions, then A trusts B [49]. The goal of trust is to “make social life predictable [by anticipating the impact of behavior], and make it easie…”
BackgroundTrust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 2×
“ay that the trust is warranted if it is the result of trustworthiness, and otherwise it is unwarranted [50]. Warranted trust is sometimes referred to as trust which is calibrated with trustworthiness [43]. In other words, trust is the cognitive mechanism to give the ‘illusion’ of anticipating intended behavior, which…”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 2×
“s can reveal strong correlations between attributes for one of those classes, giving leads to research on causation between them, even if the model was not useful for the original classification task [42, 46]. 2Assume that the performance evaluation is representative of real usage for now, although this is an importan…”
BackgroundInfluentialDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“Transparency High-standard documentation All Technical explainability Factsheets (explainability) Saliency maps [61], self-attention patterns [40], influence functions [38], probing [16] Adaptable user-centered explainability Factsheets (explainability) Counterfactual [21], contrastive [51], natural language [27], by-…”
BackgroundExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Self-citationHumans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 2×
“8We consider AI as an automation attributed with intent, and thus, Human-AI trust is a sub-case of human-machine trust [28, 55].”
BackgroundLewis, J. D., & Weigert, A. (1985) · Social Forces · doi:10.2307/2578601 · Cited by 4,163 · cited 1×
“the influence of the trustee’s incentives on Human-AI trust, should build on the existing research in sociology that investigates the influence of relationships and communities on interpersonal trust [44]. Consequently, the question of whether this trust is warranted or unwarranted is ill-defined.Our definition of war…”
BackgroundA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“We set the “effort boundary” around 774 tokens—a few of minutes of reading—and the “vulnerability boundary” right from natural language inference [7], commonsense QA [70], and multi-sentence”
BackgroundCommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1421 · Cited by 2,829 · cited 1×
“We set the “effort boundary” around 774 tokens—a few of minutes of reading—and the “vulnerability boundary” right from natural language inference [7], commonsense QA [70], and multi-sentence”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Baier, A. (1986) · Ethics · doi:10.1086/292745 · Cited by 2,567 · cited 1×
“present two directions for extension of our formalization inspired by other factors of interpersonal trust. 9.1 Other Elements of Interpersonal Trust and Trust in Automation Trust vs Reliance. Baier [4] introduces the term reliance for trust in inanimate objects, such as trusting a stool to hold your weight. We may fe…”
BackgroundInfluentialA Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
The Parable of Google Flu: Traps in Big Data Analysis
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014) · Science · doi:10.1126/science.1248506 · Cited by 2,523 · cited 1×
“the original classification task [42, 46].”
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474 · cited 1×
“rk that proposes standardized documentations to communicate the performance characteristics of trained AI models. The examples of such documentations are: data statements [6], datasheets for datasets [18], model cards [54], reproducibility checklists [56], fairness checklists [48], and factsheets [2]. We illustrate th…”
BackgroundGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · cited 1×
“rk that proposes standardized documentations to communicate the performance characteristics of trained AI models. The examples of such documentations are: data statements [6], datasheets for datasets [18], model cards [54], reproducibility checklists [56], fairness checklists [48], and factsheets [2]. We illustrate th…”
BackgroundMachine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842 · cited 1×
“With the rise of opaque and poorly-understood machine learning models in the field of AI, trust is often cited as a key desirable property of the interaction between any user and the AI [8, 11, 71, 76].”
BackgroundUnderstanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240 · cited 1×
“The recent rapid growth in the field of explainable AI (XAI) is also, in part, motivated by the need to maintain trust between the human user and the AI [31, 35, 46, 52, 59, 72].”
BackgroundAnnotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907 · cited 1×
“y and data governance Ensure privacy and data protection Datasheets/statements Removal of protected attributes [57] Ensure quality and integrity of data Datasheets/statements Detecting data artifacts [23] Establish data access protocols Datasheets/statements - Transparency High-standard documentation All - Technical e…”
BackgroundBeyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890
Bender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854 · cited 1×
“defining contracts is the work that proposes standardized documentations to communicate the performance characteristics of trained AI models. The examples of such documentations are: data statements [6], datasheets for datasets [18], model cards [54], reproducibility checklists [56], fairness checklists [48], and fact…”
BackgroundTrust in modern societies: the search for the bases of social order
(1996) · Choice Reviews Online · doi:10.5860/choice.34-1248 · Cited by 840 · cited 1×
“s vulnerability to B’s actions, then A trusts B [49]. The goal of trust is to “make social life predictable [by anticipating the impact of behavior], and make it easier to collaborate between people” [53].1 1This definition of trust is considered overly simplistic in sociology. In Section 9 we discuss aspects of more…”
BackgroundExplainable AI: A Brief Survey on History, Research Areas, Approaches and Challenges
Xu, F., Uszkoreit, H., Du, Y., Fan, W., Zhao, D., & Zhu, J. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32236-6_51 · Cited by 758 · cited 1×
“With the rise of opaque and poorly-understood machine learning models in the field of AI, trust is often cited as a key desirable property of the interaction between any user and the AI [8, 11, 71, 76].”
BackgroundA Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629
Trustworthy artificial intelligence
Thiebes, S., Lins, S., & Sunyaev, A. (2021) · Electronic Markets · doi:10.1007/s12525-020-00441-4 · Cited by 595
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564
ERASER: A Benchmark to Evaluate Rationalized NLP Models
DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.408 · Cited by 561
Trust, Distrust and Commitment
Hawley, K. (2014) · Noûs · doi:10.1111/nous.12000 · Cited by 535 · cited 2×
“at distinguish the model’s correct and incorrect cases are available to the user. 3.2 The General Case: Trust in a Contract The above example of model correctness is merely an instance of what Hawley [26], Tallant [67] refer to as trust with commitment or contractual trust. Contractual trust is a model of trust in whi…”
BackgroundInfluentialLearning The Difference That Makes A Difference With Counterfactually-Augmented Data
Kaushik, D., Hovy, E. H., & Lipton, Z. C. (2020) · International Conference on Learning Representations · Cited by 511
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
Das, A., & Rad, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11371 · Cited by 504 · cited 1×
“With the rise of opaque and poorly-understood machine learning models in the field of AI, trust is often cited as a key desirable property of the interaction between any user and the AI [8, 11, 71, 76].”
BackgroundTowards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?
Jacovi, A., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.386 · Cited by 503
Self-citationRevealing the Dark Secrets of BERT
Kovaleva, O., Romanov, A., Rogers, A., & Rumshisky, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1445 · Cited by 495 · cited 1×
“Transparency High-standard documentation All Technical explainability Factsheets (explainability) Saliency maps [61], self-attention patterns [40], influence functions [38], probing [16] Adaptable user-centered explainability Factsheets (explainability) Counterfactual [21], contrastive [51], natural language [27], by-…”
BackgroundMitigating Gender Bias in Natural Language Processing: Literature Review
Sun, T., Gaut, A., Tang, S., Huang, Y., ElSherief, M., Zhao, J., Mirza, D., Belding, E., Chang, K.-W., & Wang, W. Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1159 · Cited by 492 · cited 1×
“[38], and conceptlevel [19] explanations Make AI systems identifiable as non-human - - Diversity, non-discrimination, fairness Avoid unfair bias Fairness checklists Debiasing using data manipulation [66] Encourage accessibility and universal design - - Solicit regular feedback from stakeholders Fairness checklists - S…”
MethodMadaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436 · cited 1×
“haracteristics of trained AI models. The examples of such documentations are: data statements [6], datasheets for datasets [18], model cards [54], reproducibility checklists [56], fairness checklists [48], and factsheets [2]. We illustrate the connection between these documentations and the European requirements in Ta…”
BackgroundFactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434
GCAN: Graph-aware Co-Attention Networks for Explainable Fake News Detection on Social Media
Lu, Y.-J., & Li, C.-T. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.48 · Cited by 427
Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?Preprint
Hase, P., & Bansal, M. (2020) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2005.01831 · Cited by 396 · cited 1×
“ipation’ While not an evaluation of trust, there are methods of evaluating the ability of users to successfully anticipate the AI’s behavior, an important aspect of Human-AI trust, via simulatability [15, 25]—the ability of the user to simulate the outcome of the AI on an instance level. Thus, simulatability can serve…”
MethodExamining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
Kiritchenko, S., & Mohammad, S. (2018) · Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2005 · Cited by 384 · cited 1×
“” Previous work has used specialized test sets to verify particular contracts [5, 36, 69], or outlined methodologies for constructing such evaluation [17, 33, 58, 63].”
MethodA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C. E., & Manning, C. D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.05326 · Cited by 381 · cited 1×
“We set the “effort boundary” around 774 tokens—a few of minutes of reading—and the “vulnerability boundary” right from natural language inference [7], commonsense QA [70], and multi-sentence”
BackgroundTowards Automatic Concept-based Explanations
Ghorbani, A., Wexler, J., Zou, J., & Kim, B. (2019) · arXiv · doi:10.48550/arxiv.1902.03129 · Cited by 378
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
Khashabi, D., Chaturvedi, S., Roth, M., Upadhyay, S., & Roth, D. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1023 · Cited by 370 · cited 1×
“reading comprehension [34], as these tasks are designed only to test models’ capabilities in capturing particular linguistic phenomena, and, to the best of our knowledge, there is no undesirable event from interacting with them in the base scenario.”
BackgroundThe relationship between trust in AI and trustworthy machine learning technologies
Toreini, E., Aitken, M., Coopamootoo, K., Elliott, K., Zelaya, C. G., & van Moorsel, A. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372834 · Cited by 296 · cited 1×
“on between any user and the AI [8, 11, 71, 76]. The recent rapid growth in the field ofexplainable AI (XAI) is also, in part, motivated by the need to maintain trust between the human user and the AI [31, 35, 46, 52, 59, 72]. By designing AI that users can and will trust to interact with, the AI can be safely implemen…”
BackgroundEvaluating Gender Bias in Machine Translation
Stanovsky, G., Smith, N. A., & Zettlemoyer, L. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1164 · Cited by 292
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection
Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.647 · Cited by 287
Self-citationoLMpics-On What Language Model Pre-training Captures
Talmor, A., Elazar, Y., Goldberg, Y., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00342 · Cited by 271 · cited 1×
“fferent races, a user may deem the model more trustworthy to the contract: “I trust you to not discriminate based on race.” Previous work has used specialized test sets to verify particular contracts [5, 36, 69], or outlined methodologies for constructing such evaluation [17, 33, 58, 63]. Howcanweverifywhetheranevalua…”
MethodSelf-citationGenerating Fact Checking Explanations
Atanasova, P., Simonsen, J. G., Lioma, C., & Augenstein, I. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.656 · Cited by 269
Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?
Hase, P., & Bansal, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.491 · Cited by 254
Show Your Work: Improved Reporting of Experimental Results
Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1224 · Cited by 233 · cited 1×
“and safety Resilience to attack and security Factsheets (security) Adversarial attacks and defenses [20] Fallback plan and general safety A high level of accuracy Model cards (metrics) Reliability Factsheets (concept drift) Contrast sets [17], behavioral testing [58] Reproducibility Reproducibility checklists “Show yo…”
BackgroundDecision Making and Change in Human Affairs
(1977) · Springer eBooks · doi:10.1007/978-94-010-1276-8 · Cited by 219
Contrastive explanation: a structural-model approach
Miller, T. (2021) · The Knowledge Engineering Review · doi:10.1017/s0269888921000102 · Cited by 186 · cited 1×
“inability) Saliency maps [61], self-attention patterns [40], influence functions [38], probing [16] Adaptable user-centered explainability Factsheets (explainability) Counterfactual [21], contrastive [51], natural language [27], by-example [38], and conceptlevel [19] explanations Make AI systems identifiable as non-hu…”
BackgroundSelf-citationWhat Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models
Dalvi, F., Durrani, N., Sajjad, H., Belinkov, Y., Bau, A., & Glass, J. R. (2019) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v33i01.33016309 · Cited by 174
Hendricks, L. A., Hu, R., Darrell, T., & Akata, Z. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_17 · Cited by 171
Flores, F., & Solomon, R. C. (1998) · Business Ethics Quarterly · doi:10.2307/3857326 · Cited by 168 · cited 1×
“This perspective distinguishes “trust” (an attitude of the trustor) from being “trustworthy” (a property of the trustee) [64, 75], and we say that an AI model is trustworthy to some contract if it is capable of maintaining this contract.”
BackgroundLet Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems
Kunkel, J., Donkers, T., Michael, L., Barbu, C.-M., & Ziegler, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300717 · Cited by 156 · cited 1×
“oversight Foster fundamental human rights Fairness checklists See “Diversity, non-discrimination, fairness” Support users’ agency All User-centered explanations [59] Enable human oversight Explanations in recommender systems [41]”
BackgroundProbing for semantic evidence of composition by means of simple classification tasks
Ettinger, A., Elgohary, A., & Resnik, P. (2016) · Workshop on Evaluating Vector-Space Representations for NLP · doi:10.18653/v1/w16-2524 · Cited by 147 · cited 1×
“ts/statements - Transparency High-standard documentation All - Technical explainability Factsheets (explainability) Saliency maps [61], self-attention patterns [40], influence functions [38], probing [16] Adaptable user-centered explainability Factsheets (explainability) Counterfactual [21], contrastive [51], natural…”
BackgroundAligning Faithful Interpretations with their Social Attribution
Jacovi, A., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00367 · Cited by 137 · cited 1×
“fails us, but this is not true for reliance because we did not attribute intent to the trustee [68]. Despite the fact that AI models are tools, people anthropomorphize them and assign intent to them [30, 52]. This makes us believe that reliance is not suitable for AI models, and for this reason we aim to define Human-…”
BackgroundSelf-citationCan we do better explanations? A proposal of user-centered explainable AI
Ribera, M., & Lapedriza, À. (2019) · IUI Workshops · Cited by 133
Smith-Renner, A., Fan, R., Birchfield, M., Wu, T., Boyd-Graber, J., Weld, D. S., & Findlater, L. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376624 · Cited by 112 · cited 1×
“al setup to evaluate unwarranted trust, and conclude that even data scientists are susceptible to develop unwarranted trust, despite some mathematical understanding of the models. Smith-Renner et al. [62] show similar conclusions on unwarranted trust in a different experimental setup. However, this area is underdevelo…”
BackgroundAn Interpretable Model with Globally Consistent Explanations for Credit RiskPreprint
Chen, C., Lin, K., Rudin, C. D., Shaposhnik, Y., Wang, S., & Wang, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12615 · Cited by 108 · cited 1×
“For example, AI-produced credit scoring [9] represents a risk to the loan officer: a wrong decision carries a risk (among others) that the applicant defaults in the future.”
BackgroundWright, S. (2009) · Philosophia · doi:10.1007/s11406-009-9218-0 · Cited by 89 · cited 1×
“This perspective distinguishes “trust” (an attitude of the trustor) from being “trustworthy” (a property of the trustee) [64, 75], and we say that an AI model is trustworthy to some contract if it is capable of maintaining this contract.”
BackgroundMarasović, A., Bhagavatula, C., Park, J. S., Bras, R. J. L., Smith, N. A., & Choi, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.253 · Cited by 78
Self-citationTowards Ecologically Valid Research on Language User InterfacesPreprint
de Vries, H., Bahdanau, D., & Manning, C. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14435 · Cited by 74 · cited 1×
“ome specific way, for which gold labels are available. Test sets are generally seen as imperfect approximators of post-deployment data, as their distribution is not representative of true unseen data [12] and thus cannot imply overall generalization. However, this perspective only takes into account one type of contra…”
BackgroundEvaluating NLP Models via Contrast SetsPreprint
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · Cited by 65
Targeting the Benchmark: On Methodology in Current Natural Language Processing Research
Schlangen, D. (2021) · doi:10.18653/v1/2021.acl-short.85 · Cited by 60 · cited 1×
“Motivated by categorization in [63], we set the “vulnerability boundary”—a hypothetical threshold for tasks without an associated undesirable event— right from natural language inference [7],”
MethodAsk Not What AI Can Do, But What AI Should Do: Towards a Framework of Task DelegabilityPreprint
Lubars, B., & Tan, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.03245 · Cited by 48
Sentiment Analysis Is Not Solved! Assessing and Probing Sentiment Classification
Barnes, J., Øvrelid, L., & Velldal, E. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4802 · Cited by 43 · cited 1×
“” Previous work has used specialized test sets to verify particular contracts [5, 36, 69], or outlined methodologies for constructing such evaluation [17, 33, 58, 63].”
MethodSimplicity Creates Inequity: Implications for Fairness, Stereotypes, and Interpretability
Kleinberg, J. M., & Mullainathan, S. (2019) · ACM Conference on Economics and Computation (EC) · doi:10.1145/3328526.3329621 · Cited by 37 · cited 1×
“Accountability Auditability of algorithms/data/design Factsheets (lineage) Minimize and report negative impacts Fairness checklists Acknowledge and evaluate trade-offs Reporting the robustness-accuracy [1] trade-off or simplicity-equity trade-off [37] Ensure redress Fairness checklists -”
BackgroundClinicalVis: Supporting Clinical Task-Focused Design Evaluation
Ghassemi, M., Pushkarna, M., Wexler, J., Johnson, J., & Varghese, P. (2018) · arXiv · doi:10.48550/arxiv.1810.05798 · Cited by 26 · cited 1×
“[19] show a case where the interface can increase doctors’ confidence in a tool, despite not significantly increasing their accuracy.”
BackgroundCommitment in Cases of Trust and Distrust
Tallant, J. (2017) · Thought A Journal of Philosophy · doi:10.1002/tht3.259 · Cited by 26
Algorithmic Fairness: Measures, Methods and Representations
Venkatasubramanian, S. (2019) · ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems (PODS) · doi:10.1145/3294052.3322192 · Cited by 22 · cited 1×
“For example, there are multiple formal measures of fairness such as individual fairness, demographic parity and equalized error rates [78].”
BackgroundShow Your Work: Improved Reporting of Experimental Results
Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019) · arXiv · doi:10.48550/arxiv.1909.03004 · Cited by 22 · cited 1×
“and safety Resilience to attack and security Factsheets (security) Adversarial attacks and defenses [20] Fallback plan and general safety A high level of accuracy Model cards (metrics) Reliability Factsheets (concept drift) Contrast sets [17], behavioral testing [58] Reproducibility Reproducibility checklists “Show yo…”
BackgroundTrust: from the Philosophical to the Commercial
Tallant, J., & Donati, D. (2019) · Philosophy of Management · doi:10.1007/s40926-019-00107-y · Cited by 18
Intrinsic and Enforceable Trust: A Research Agenda
Hofstede, G. J., & Hofstede, G. J. (2006) · AgEcon Search (University of Minnesota, USA) · doi:10.22004/ag.econ.7769 · Cited by 13
The International Convention Centre
Smyth, H. (2005) · Taylor & Francis eBooks · doi:10.4324/9780203975954-14 · Cited by 11
Research works printed but not linked: 27
- David Alvarez Melis and Tommi Jaakkola. 2018. Towards Robust Interpretability with Self-Explaining Neural Networks. In Advances in Neural Information Processing Systems 31 S. Bengio H. Wallach H. Larochelle K. Grauman N. Cesa-Bianchi and R. Garnett (Eds.). 7775--7784. http://papers.nips.cc/paper/8003-towards-robust-interpretability-with-self-explaining-neural-networks.pdf
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 (2017). · arXiv
- 10.18653/v1 · doi:10.18653/v1
- Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1412.6572 · arXiv
- Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Counterfactual Visual Explanations (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, Long Beach, California, USA, 2376--2384. http://proceedings.mlr.press/v97/goyal19a.html
- 10.18653/v1 · doi:10.18653/v1
- Sven Ove Hansson. 2018. Risk. In The Stanford Encyclopedia of Philosophy (fall 2018 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.
- Robert R Hoffman. 2017. A taxonomy of emergent trusting in the human-machine relationship. Cognitive systems engineering: The future for a changing world (2017), 137--163.
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- Hana Kopecka and Jose M Such. 2020. Explainable AI for Cultural Minds. https://sites.google.com/view/dexahai-at-ecai2020/home Workshop on Dialogue, Explanation and Argumentation for Human-Agent Interaction, DEXAHAI; Conference date: 07-09-2020.
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- Carolyn McLeod. 2015. Trust. In The Stanford Encyclopedia of Philosophy (fall 2015 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- Philipp Schmidt and Felix Bießmann. 2019. Quantifying Interpretability and Trust in Machine Learning Systems. CoRR abs/1901.08558 (2019). arXiv:1901.08558 http://arxiv.org/abs/1901.08558 · arXiv
- Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. 2020. We Need to Talk About Random Splits. CoRR abs/2005.00636 (2020). arXiv:2005.00636 https://arxiv.org/abs/2005.00636 · arXiv
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Algorithmic fairness in artificial intelligence for medicine and healthcare
Chen, R. J., Wang, J. J., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Lu, M. Y., Sahai, S., & Mahmood, F. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01056-8 · Cited by 750
“Such contracts are AI functionalities that are anticipated to have known vulnerabilities 305 .”
BackgroundTrustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
Trust in AI: progress, challenges, and future directions
Afroogh, S., Akbari, A., Malone, E., Kargar, M., & Alambeigi, H. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-04044-8 · Cited by 507
“In Jacovi et al. (2021b), a precise discussion is presented regarding the nature of trust in AI, as well as the prerequisites and goals of the cognitive mechanism of trust.”
InfluentialUnderspecification Presents Challenges for Credibility in Modern Machine Learning
D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. (2020) · arXiv · doi:10.48550/arxiv.2011.03395 · Cited by 398
“Strategies of this type are one way to warrant trust in an ML system (Jacovi et al., 2020).”
Background
Metadata
- Authors
- Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y.
- Affiliations
- Bar-Ilan University (IL)
- University of Washington (US)
- The University of Melbourne (AU)
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 624-635
- OpenAlex
- W3134111219
- Cited by
- 545
- References
- 74
- Citations by year
- 2026: 131 · 2025: 137 · 2024: 102 · 2023: 90 · 2022: 60 · 2021: 24
- FWCI
- 31.88
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- Funders
- European Research Council (802774)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- trust in AI, interpersonal trust, contractual trust, trustworthiness, user vulnerability, explainable AI, human-AI interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.