Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
The DOI matches its Crossref record; 74 of the 76 references it cites were found in a registry or an index, and 2 are not checked yet.
Abstract
Trust is a central component of the interaction between people and AI, in that 'incorrect' levels of trust may cause misuse, abuse or disuse of the technology. But what, precisely, is the nature of trust in AI? What are the prerequisites and goals of the cognitive mechanism of trust, and how can we promote them, or assess whether they are being satisfied in a given interaction? This work aims to answer these questions. We discuss a model of trust inspired by, but not identical to, interpersonal trust (i.e., trust between people) as defined by sociologists. This model rests on two key properties: the vulnerability of the user; and the ability to anticipate the impact of the AI model's decisions. We incorporate a formalization of 'contractual trust', such that trust between a user and an AI model is trust that some implicit or explicit contract will hold, and a formalization of 'trustworthiness' (that detaches from the notion of trustworthiness in sociology), and with it concepts of 'warranted' and 'unwarranted' trust. We present the possible causes of warranted trust as intrinsic reasoning and extrinsic behavior, and discuss how to design trustworthy AI, how to evaluate whether trust has manifested, and whether it is warranted. Finally, we elucidate the connection between trust and XAI using our formalization.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Uncertainty is a defining characteristic of trust (Mayer et al., 1995) and has been referred to as a necessary prerequisite of human trust in AI that has been lacking in current XAI studies (Jacovi et al., 2021).”
From Exploring the effects of human-centered AI explanations on trust and reliance (2023) · cites this paper as Jacovi et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“research found that people often ignore recommendations because they do not trust them”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The primary motivation for clinical XAI is fostering appropriate trust: enabling clinicians to calibrate their reliance on AI recommendations based on the quality of supporting reasoning [156,157].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [157]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, people find it difficult to correctly calibrate their trust in a decision aid [10,13,21,37].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“For example, [38] formalizes human-AI trust by considering and extending the concepts of interpersonal trust.”
From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [38]
9 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736 · cited 1×
“on basic definition of trust regards it as a directional transaction between two parties: if A believes that B will act in A’s best interest, and accepts vulnerability to B’s actions, then A trusts B [49]. The goal of trust is to “make social life predictable [by anticipating the impact of behavior], and make it easie…”
BackgroundTrust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 2×
“ay that the trust is warranted if it is the result of trustworthiness, and otherwise it is unwarranted [50]. Warranted trust is sometimes referred to as trust which is calibrated with trustworthiness [43]. In other words, trust is the cognitive mechanism to give the ‘illusion’ of anticipating intended behavior, which…”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 2×
“s can reveal strong correlations between attributes for one of those classes, giving leads to research on causation between them, even if the model was not useful for the original classification task [42, 46]. 2Assume that the performance evaluation is representative of real usage for now, although this is an importan…”
BackgroundInfluentialDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“Transparency High-standard documentation All Technical explainability Factsheets (explainability) Saliency maps [61], self-attention patterns [40], influence functions [38], probing [16] Adaptable user-centered explainability Factsheets (explainability) Counterfactual [21], contrastive [51], natural language [27], by-…”
BackgroundExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Self-citationHumans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 2×
“8We consider AI as an automation attributed with intent, and thus, Human-AI trust is a sub-case of human-machine trust [28, 55].”
BackgroundLewis, J. D., & Weigert, A. (1985) · Social Forces · doi:10.2307/2578601 · Cited by 4,163 · cited 1×
“the influence of the trustee’s incentives on Human-AI trust, should build on the existing research in sociology that investigates the influence of relationships and communities on interpersonal trust [44]. Consequently, the question of whether this trust is warranted or unwarranted is ill-defined.Our definition of war…”
BackgroundA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“We set the “effort boundary” around 774 tokens—a few of minutes of reading—and the “vulnerability boundary” right from natural language inference [7], commonsense QA [70], and multi-sentence”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Baier, A. (1986) · Ethics · doi:10.1086/292745 · Cited by 2,567 · cited 1×
“present two directions for extension of our formalization inspired by other factors of interpersonal trust. 9.1 Other Elements of Interpersonal Trust and Trust in Automation Trust vs Reliance. Baier [4] introduces the term reliance for trust in inanimate objects, such as trusting a stool to hold your weight. We may fe…”
BackgroundInfluential
Research works printed but not linked: 27
- [1] David Alvarez Melis and Tommi Jaakkola. 2018. Towards Robust Interpretability with Self-Explaining Neural Networks. In Advances in Neural Information Processing Systems 31 , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett (Eds.). 7775-7784. http://papers.nips.cc/paper/8003towards-robust-interpretability-with-self-explaining-neural-networks.pdf -
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- [15] Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 (2017). - · arXiv
- 10.18653/v1 · doi:10.18653/v1
- [21] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1412.6572 - · arXiv
- [22] Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Counterfactual Visual Explanations (Proceedings of Machine Learning Research, Vol. 97) , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, Long Beach, California, USA, 2376-2384. http://proceedings.mlr.press/v97/goyal19a.html -
- 10.18653/v1 · doi:10.18653/v1
- [25] Sven Ove Hansson. 2018. Risk. In The Stanford Encyclopedia of Philosophy (fall 2018 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University. -
- [29] Robert R Hoffman. 2017. A taxonomy of emergent trusting in the human-machine relationship. Cognitive systems engineering: The future for a changing world (2017), 137-163. -
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- [40] Hana Kopecka and Jose M Such. 2020. Explainable AI for Cultural Minds. https://sites.google.com/view/dexahai-at-ecai2020/home Workshop on Dialogue, Explanation and Argumentation for Human-Agent Interaction , DEXAHAI ; Conference date: 07-09-2020. -
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- [53] Carolyn McLeod. 2015. Trust. In The Stanford Encyclopedia of Philosophy (fall 2015 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University. -
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- [64] Philipp Schmidt and Felix Bießmann. 2019. Quantifying Interpretability and Trust in Machine Learning Systems. CoRR abs/1901.08558 (2019). arXiv:1901.08558 http://arxiv.org/abs/1901.08558 - · arXiv
- [67] Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. 2020. We Need to Talk About Random Splits. CoRR abs/2005.00636 (2020). arXiv:2005.00636 https://arxiv.org/abs/2005.00636 - · arXiv
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
Cited by
Items 101–200 of 747
Influences on User Trust in Healthcare Artificial Intelligence: A Systematic Review
Jermutus, E., Kneale, D., Thomas, J., & Michie, S. (2022) · Wellcome Open Research · doi:10.12688/wellcomeopenres.17550.1 · Cited by 44
“The majority of the papers (18) were excluded after reading the full text as their research design did not match the required criterion, i.”
BackgroundVereschak, O., Alizadeh, F., Bailly, G., & Caramiaux, B. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642018 · Cited by 72
A Project Based Learning Approach for Improving Students’ Computational Thinking Skills
Bertacchini, F., Scuro, C., Pantano, P., & Bilotta, E. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.720448 · Cited by 52
Unpacking Human-AI Interaction in Safety-Critical Industries: A Systematic Literature Review
Bach, T. A., Kristiansen, J. K., Babić, A., & Jacovi, A. (2024) · IEEE Access · doi:10.1109/access.2024.3437190 · Cited by 62
“It may be best to focus research on calibrating HAII to foster the appropriate level of trust and/or reliance (i.e., ‘‘warranted trust’’ [117]) in the output of AI-enabled systems during a decision-making process [39], [116].”
BackgroundSelf-citationThe key role of design and transparency in enhancing trust in AI-powered digital agents
Glassberg, I., Brender‐Ilan, Y., & Zwilling, M. (2025) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2025.100770 · Cited by 51
The Robustness of Counterfactual Explanations Over Time
Ferrario, A., & Loi, M. (2022) · IEEE Access · doi:10.1109/access.2022.3196917 · Cited by 64
Cheng, R., Wang, R., Zimmermann, T., & Ford, D. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3651990 · Cited by 47
“Therefore, in addition to making technologies more trustworthy, it is also important to support users in deciding when and how much to trust an AI, so that they can use their expertise when needed to complement the capabilities of AI systems [ 28 , 34 ].”
BackgroundMethodInfluentialEisbach, S., Langer, M., & Hertel, G. (2023) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2023.100015 · Cited by 46
When AI Fails, Who Do We Blame? Attributing Responsibility in Human–AI Interactions
Schoenherr, J. R., & Thomson, R. (2024) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2024.3370095 · Cited by 53
“The degree of trust is dependent on many factors, including whether the AI-enabled system is physically instantiated (e.g., a robot) or an algorithm, whether the task is cognitively demanding [20], and whether the environment or consequences of task failure are deemed risky [21].”
BackgroundHuang, K., & Ball, C. (2024) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1146 · Cited by 46
Hou, T.-Y., Tseng, Y.-C., & Yuan, C. W. (2024) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2024.102775 · Cited by 60
Public Opinion toward Artificial Intelligence
Zhang, B. (2023) · The Oxford Handbook of AI Governance · doi:10.1093/oxfordhb/9780197579329.013.36 · Cited by 34
Applications of Self-Supervised Learning to Biomedical Signals: A Survey
Pup, F. D., & Atzori, M. (2023) · IEEE Access · doi:10.1109/access.2023.3344531 · Cited by 43
“…of deep learning in the biomedical domain (where it’s more difficult to get data and annotations by experts) more substantial and to help face some open challenges (e.g., accountability, distribution shifts, robustness), which still hinder the reliability of AI for healthcare [202], [203].”
BackgroundUncertainty in XAI: Human Perception and Modeling Approaches
Chiaburu, T., Haußer, F., & Bießmann, F. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6020055 · Cited by 53
Learning to run a power network with trust
Marot, A., Donnot, B., Chaouache, K., Kelly, A., Huang, Q., Hossain, R. R., & Cremer, J. L. (2022) · Electric Power Systems Research · doi:10.1016/j.epsr.2022.108487 · Cited by 55
Understanding Disengagement in Just-in-Time Mobile Health Interventions
Park, J., & Lee, U. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3596240 · Cited by 48
Trust Development and Repair in AI-Assisted Decision-Making during Complementary Expertise
Pareek, S., Velloso, E., & Gonçalves, J. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658924 · Cited by 54
To Trust or Distrust Trust Measures: Validating Questionnaires for Trust in AIPreprint
Scharowski, N., Perrig, S. A. C., Aeschbach, L. F., von Felten, N., Opwis, K., Wintersberger, P., & Brühlmann, F. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2403.00582 · Cited by 45
Salimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41
“Prior work has highlighted that appropriate trust and reliance play a critical role in achieving complementary human-AI team performance [58, 90, 139, 141].”
BackgroundMethodConcerns about the role of artificial intelligence in journalism, and media manipulation
Mahony, S., & Chen, Q.-H. (2024) · Journalism · doi:10.1177/14648849241263293 · Cited by 49
“Hence the need for what has become known as explainable AI ; ‘ in part motivated by the need to maintain trust between the human user and AI ’ (Jacovi et al., 2021: 624).”
Duan, W., Zhou, S., Scalia, M. J., Freeman, G., Gorman, J. C., Tolston, M. T., McNeese, N. J., & Funke, G. J. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108560 · Cited by 48
Ethics of artificial intelligence in prenatal and pediatric genomic medicine
Coghlan, S., Gyngell, C., & Vears, D. F. (2023) · Journal of Community Genetics · doi:10.1007/s12687-023-00678-4 · Cited by 46
“The final prominent concept in AI ethics relevant to medical AI is trust (Jacovi et al. 2021).”
BackgroundInfluentialVianello, A., Laine, S., & Tuomi, E. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2095478 · Cited by 46
“In this article, we consider human-AI trust as a contractual phenomenon, as formalized by Jacovi et al. (2021).”
BackgroundResultInfluentialLanguage Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38
Zhang, R., Flathmann, C., Musick, G., Schelble, B. G., McNeese, N. J., Knijnenburg, B. P., & Duan, W. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3635474 · Cited by 44
“Previous work has studied various factors that impact team performance, such as human-AI team composition [26, 33, 68, 70], AI’s performance or features [20, 34], and human perceptions of AI teammates [71, 85, 110].”
BackgroundInfluentialTrusting Autonomous Teammates in Human-AI Teams - A Literature Review
Duan, W., Flathmann, C., McNeese, N. J., Scalia, M. J., Zhang, R., Gorman, J. C., Freeman, G., Zhou, S., Hauptman, A. I., & Yin, X. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713527 · Cited by 50
Balayn, A., Yurrita, M., Rancourt, F., Casati, F., & Gadiraju, U. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713787 · Cited by 50
“…about the AI system (e.g., discrimination) [98] and the ways they are handled by the system producer [5] in terms of ability; explanations provided for each AI output [64] in terms of integrity; transparency about the AI system [13, 39] and designed user-interaction [5, 64] in terms of benevolence.”
InfluentialExplainability Auditing for Intelligent Systems: A Rationale for Multi-Disciplinary Perspectives
Langer, M., Baum, K. L., Hartmann, K., Hessel, S., Speith, T., & Wahl, J. (2021) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew53955.2021.00030 · Cited by 45
“2) Psychological Benefits: First, ensuring the psychological dimensions may help to develop warranted trust in systems [12], [28].”
Humans, AI, and Context: Understanding End-Users’ Trust in a Real-World Computer Vision Application
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3593978 · Cited by 37
“[15] focus on formalizing prerequisites, causes, and goals of trust in AI, and Liao and Sundar [29] focus on modeling the communication of trustworthiness.”
BackgroundInfluentialRamezani, E. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129862 · Cited by 41
Shen, M. W. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.05302 · Cited by 43
“Several recent treatments on human-AI trust begin from interpersonal (human-human) trust frameworks [39, 76].”
BackgroundMethodInfluentialTech Giants’ Responsible Innovation and Technology Strategy: An International Policy Review
Li, W., Yiğitcanlar, T., Nili, A., & Browne, W. N. (2023) · Smart Cities · doi:10.3390/smartcities6060153 · Cited by 45
Towards Ethical Data-Driven Software: Filling the Gaps in Ethics Research & Practice
Johnson, B., & Smith, J. (2021) · Workshop on Ethics in Software Engineering Research and Practice · doi:10.1109/sethics52569.2021.00011 · Cited by 47
“Similarly, Jacovi and colleagues explore the formalization of trust in AI systems, but via a model of trust that is inspired by social theory on trust between people [55].”
BackgroundA Review of the Role of Causality in Developing Trustworthy AI SystemsPreprint
Ganguly, N., Fazlija, D., Badar, M., Fisichella, M., Sikdar, S., Schrader, J., Wallat, J., Rudra, K., Koubarakis, M., Patro, G. K., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.06975 · Cited by 46
“This is corroborated by several recent surveys and reports [34, 110, 118, 247, 283].”
ResultThe Importance of Distrust in AI
Peters, T. M., & Visser, R. W. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44070-0_15 · Cited by 44
Requirements Engineering in Machine Learning Projects
Gjorgjevikj, A., Mishev, K., Antovski, L., & Trajanov, D. (2023) · IEEE Access · doi:10.1109/access.2023.3294840 · Cited by 38
“Furthermore, Jacovi et al. [58] indicate that adequate verification of the existence of a certain risk (an undesirable but possible event) from the use of an AI system is a prerequisite for verification of the existence of Human-AI trust.”
BackgroundAppelganc, K., Rieger, T., Roesler, E., & Manzey, D. (2022) · Journal of Cognitive Engineering and Decision Making · doi:10.1177/15553434221104615 · Cited by 36
“That is because a perceived vulnerability and therefore differences in risk for a certain outcome can in fl uence trust (Jacovi et al., 2021).”
BackgroundInfluentialDuan, X., Pei, B., Ambrose, G. A., Hershkovitz, A., Cheng, Y., & Wang, C. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11954-8 · Cited by 35
“Jacovi et al. ( 2021 ), focusing on individual interactions with AI, defined the major properties of trust as the vulnerability of the user and the ability to anticipate the impact of models’ decisions.”
Trust, artificial intelligence and software practitioners: an interdisciplinary agenda
Pink, S., Quilty, E., Grundy, J., & Hoda, R. (2024) · AI & Society · doi:10.1007/s00146-024-01882-7 · Cited by 39
Chaturvedi, A., Yadav, N., & Dasgupta, M. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2456534 · Cited by 35
“Trust in AI is another prominent dependent variable and relates to the manager’s firm belief that they have the capability of AI (Jacovi et al., 2021).”
Chalhoub, G., & Sarkar, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501833 · Cited by 38
“Trust is a well-documented challenge of interaction with intelligent systems [56, 86].”
BackgroundBetween transparency and trust: identifying key factors in AI system perception
Cetinkaya, N. E., & Krämer, N. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2533358 · Cited by 44
“A significant body of research highlights that explainability is crucial for establishing AI trustworthiness (Caspers 2021; Jacovi et al. 2021).”
Kaate, I., Salminen, J. O., Jung, S., Almerekhi, H., & Jansen, B. J. J. (2023) · Biannual Conference of the Italian SIGCHI Chapter (CHItaly) · doi:10.1145/3605390.3605397 · Cited by 45
“Our study adds to prior research in which similar themes have been studied on human likeness [30], trust [20, 27], glitches and distortions [29], and empathy (i.”
ResultRedesigning Primary Care: The Emergence of Artificial-Intelligence-Driven Symptom Diagnostic Tools
Wiedermann, C. J., Mahlknecht, A., Piccoliori, G., & Engl, A. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13091379 · Cited by 37
“Trust in symptom checkers might differ between patients and GPs based on their respective vulnerabilities, the anticipation of the AI’s decisions, an understanding of the ‘contract’ with the AI, an evaluation of the AI’s trustworthiness, and what is needed from explainable AI [21].”
BackgroundFlora, M. L., Potvin, C. K., McGovern, A., & Handler, S. L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.08943 · Cited by 41
“As part of building trust with human forecasters in ML predictions, it is important to explain the "why" of an ML model’s prediction in understandable terms and to create real-time visualizations of these methods (Hoffman et al. 2017; Karstens et al. 2018; Jacovi et al. 2021).”
Pelly, M. L., Fatehi, F., Liew, D., & Verdejo‐García, A. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105041 · Cited by 31
Calibrating workers’ trust in intelligent automated systems
Lucas, G., Becerik-Gerber, B., & Roll, S. C. (2024) · Patterns · doi:10.1016/j.patter.2024.101045 · Cited by 36
How Learners’ Trust Changes in Generative AI over a Semester of Undergraduate Courses
Song, D. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00446-6 · Cited by 32
“The pivotal role of trust in shaping interactions between individuals and AI can - not be overstated, as inappropriate levels of trust can result in misuse, exploitation, or neglect of the technology (Jacovi et al., 2021).”
Should Users Trust Advanced AI Assistants? Justified Trust As a Function of Competence and Alignment
Manzini, A., Keeling, G., Marchal, N., McKee, K. R., Rieser, V., & Gabriel, I. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658964 · Cited by 28
“…justified user trust in AI assistants, appropriate and robust evaluations need to pay particular attention to the way in which users interact with this technology and the impact that such interactions have on users (see [53] for a critical discussion of what evaluation of trust should require).”
BackgroundPerceiving AI intervention does not compromise the persuasive effect of fact-checking
Chae, J. H., & Tewksbury, D. (2024) · New Media & Society · doi:10.1177/14614448241286881 · Cited by 40
“This suggests that, despite widespread discussions about public distrust in AI (Dwork and Minow, 2022; Jacovi et al., 2021), at least in the context of fact-checking, people’s perceptions of AI’s role in information generation do not seem to be a significant concern.”
Artificial Intelligence Project Success Factors—Beyond the Ethical Principles
Miller, G. J. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-030-98997-2_4 · Cited by 41
Schrills, T., & Franke, T. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3588594 · Cited by 32
“In this context, clinical reviews, as required from professionals in studies regarding medical AI systems [ 48 ], are one way to provide evidence of trustworthiness and thus increase “extrinsic trust” [ 56 ].”
BackgroundKaate, I., Salminen, J. O., Santos, J. M., Jung, S., Almerekhi, H., & Jansen, B. J. J. (2023) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2023.100031 · Cited by 31
What’s the Appeal? Perceptions of Review Processes for Algorithmic Decisions
Lyons, H., Wijenayake, S., Miller, T., & Velloso, E. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517606 · Cited by 27
Self-citationUse of a large language model (LLM) for ambulance dispatch and triage
Shekhar, A. C., Kimbrell, J. M., Saharan, A., Stebel, J., Ashley, E. R., & Abbott, E. E. (2024) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2024.12.032 · Cited by 32
Balancing AI transparency: Trust, Certainty, and Adoption
Ngo, V. M. (2025) · Information Development · doi:10.1177/02666669251346124 · Cited by 30
Building Trust in Artificial Intelligence: A Systematic Review through the Lens of Trust Theory
Regona, M., Yiğitcanlar, T., Hon, C. K. H., & Teo, M. M. (2026) · ACM Computing Surveys · doi:10.1145/3789256 · Cited by 33
Osman, K. A. Y. A., Shah, A. F. M. S., Karabulut, M. A., Karahan, S. N., Osmanca, M. S., & Acır, N. (2026) · IEEE Access · doi:10.1109/access.2026.3663161 · Cited by 37
“Actionable explanations extend further by providing concrete guidance for achieving different outcomes, such as specifying that ‘‘increasing annual income by $10,000 would significantly improve loan approval likelihood’’ [89].”
A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI)
Speith, T., & Langer, M. (2023) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew57809.2023.00061 · Cited by 35
Leveraging Chatbots to Combat Health Misinformation for Older Adults: Participatory Design Study
Peng, W., Lee, H. R., & Lim, S. (2024) · JMIR Formative Research · doi:10.2196/60712 · Cited by 33
Human trust in AI: a relationship beyond reliance
Blanco, S. (2025) · AI and Ethics · doi:10.1007/s43681-025-00690-z · Cited by 34
Suresh, H., Lewis, K. M., Guttag, J., & Satyanarayan, A. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511160 · Cited by 20
“Indeed, research on human trust suggests that in addition to conveying assurances of certainty, acknowledging when systems are uncertain is also an important factor in building effective trust [35].”
BackgroundMethodSpotting When Algorithms Are Wrong
Buijsman, S., & Veluwenkamp, H. (2022) · Minds and Machines · doi:10.1007/s11023-022-09591-0 · Cited by 29
“Of course, under exactly which conditions users are justified to trust a (ML) system is an open research question (cf. Durán & Jongsma, 2021; Ferrario, 2021; Ferrario & Loi, 2021; Jacovi et al., 2021) and so this assumption may be wrong.”
To Trust or Distrust AI: A Questionnaire Validation Study
Scharowski, N., Perrig, S. A. C., von Felten, N., Aeschbach, L. F., Opwis, K., Wintersberger, P., & Brühlmann, F. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732025 · Cited by 37
“They refer to warranted trust as trust calibrated with trustworthiness [35].”
InfluentialMcGovern, A., Bostrom, A., Davis, P., Demuth, J. L., Ebert‐Uphoff, I., He, R., Hickey, J. J., Gagne, D. J., Snook, N., Stewart, J. Q., et al. (2022) · Bulletin of the American Meteorological Society · doi:10.1175/bams-d-21-0020.1 · Cited by 26
“…bear risk analysis, perception, and communication research theories and methods, with the goals of understanding how different features of AI and XAI influence trustworthiness of, trust in, and willingness to use AI guidance (e.g., Jacovi et al. 2021; Mueller et al. 2019; Glikson and Woolley 2020).”
BackgroundTrusting RoBERTa over BERT: Insights from CheckListing the Natural Language Inference TaskPreprint
Tarunesh, I., Aditya, S., & Choudhury, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.07229 · Cited by 30
“Following the recent XAI literature (Jacovi et al., 2021), we aim to quantify progress towards more predictable natural language understanding models (especially PTLMs).”
BackgroundMethodYoo, D. W., Woo, H., Nguyen, V. C., Birnbaum, M. L., Kruzan, K. P., Kim, J. G., Abowd, G. D., & Choudhury, M. D. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642369 · Cited by 28
Lassiter, T. B., & Fleischmann, K. R. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686963 · Cited by 26
“It is often argued that trust can only exist if there is vulnerability as there must be some risk in trusting [34].”
(Re)Conceptualizing trustworthy AI: A foundation for change
Wirz, C. D., Demuth, J. L., Bostrom, A., Cains, M. G., Ebert‐Uphoff, I., Gagne, D. J., Schumacher, A., McGovern, A., & Madlambayan, D. (2025) · Artificial Intelligence · doi:10.1016/j.artint.2025.104309 · Cited by 31
Selective Trust: Understanding Human-AI Partnerships in Personal Health Decision-Making Process
van Arum, S., Genç, H. U., Reidsma, D., & Karahanoğlu, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713462 · Cited by 31
Dalvi‐Esfahani, M., ahmadabadi, H. B., Ramayah, T., Turner, J. J., Iahad, N. A., & Azar, N. (2025) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2025.124291 · Cited by 27
Trustworthy AI and the Logics of Intersectional Resistance
Knowles, B., Fledderjohann, J., Richards, J. T., & Varshney, K. R. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593986 · Cited by 30
Feldhus, N., Hennig, L., Nasert, M. D., Ebert, C., Schwarzenberg, R., & Mller, S. (2023) · doi:10.18653/v1/2023.nlrse-1.4 · Cited by 36
“Alignment of model rationales with very few human-acceptable gold rationales may raise issues of trust (Jacovi et al., 2021) and the models trained on them may suffer from hallucinations (Maynez et al., 2020).”
BackgroundHe, G., Demartini, G., & Gadiraju, U. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713218 · Cited by 23
“Users with an uncalibrated trust in the AI system can be easily misled to disuse or misuse AI systems [42].”
Garcia, K. R., Mishler, S., Xiao, Y., Wang, C., Hu, B., Still, J. D., & Chen, J. (2022) · Journal of Cognitive Engineering and Decision Making · doi:10.1177/15553434221117001 · Cited by 25
“Similar to interpersonal trust (i.e., trust between people), human-AI trust focuses on the vulnerability of the human user and their ability to anticipate the outcome of the AI ’ s decision (Jacovi et al., 2021).”
BackgroundWould you trust an AI chef? Examining what people think when AI becomes creative with food
Califano, G., Zhang, T., & Spence, C. J. (2024) · International Journal of Gastronomy and Food Science · doi:10.1016/j.ijgfs.2024.100973 · Cited by 24
Introducing contextual transparency for automated decision systems
Sloane, M., Solano-Kamaiko, I. R., Yuan, J., Dasgupta, A., & Stoyanovich, J. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00623-7 · Cited by 28
“https://doi.org/10.1038/s42256-023-00623-7 questions for creating mechanisms that foster ADS interpretability and trust 37 : what are we explaining?”
BackgroundInfluentialChun, J., & Elkins, K. (2023) · International Journal of Digital Humanities · doi:10.1007/s42803-023-00069-8 · Cited by 34
Designing and Evaluating User Experience of an AI-Based Defense System
Park, S., Kim, H. K., Park, J., & Lee, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3329257 · Cited by 25
Scherer, B., & Lehner, S. (2021) · Journal of Empirical Finance · doi:10.1016/j.jempfin.2024.101574 · Cited by 23
Chen, C., Lee, S., Jang, E., & Sundar, S. S. (2024) · Proceedings of the Second International Symposium on Trustworthy Autonomous Systems · doi:10.1145/3686038.3686060 · Cited by 21
The Trust Recovery Journey. The Effect of Timing of Errors on the Willingness to Follow AI Advice.
Kahr, P. K., Rooks, G., Snijders, C., & Willemsen, M. C. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645167 · Cited by 21
Correctness is not Faithfulness in Retrieval Augmented Generation Attributions
Wallat, J., Heuss, M., Rijke, M. D., & Anand, A. (2025) · Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) · doi:10.1145/3731120.3744592 · Cited by 21
Robertson, S., & Díaz, M. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3534638 · Cited by 31
“[33] describe distrust in AI as a mode of mitigating risk, which must be present for trust or distrust to manifest [39], but it is unclear how MT users with limited language abilities assess and mitigate the risk of inadequate translations, or calibrate their trust in MT.”
BackgroundScherer, B., & Lehner, S. (2022) · Journal of Asset Management · doi:10.1057/s41260-022-00284-y · Cited by 21
Establishing Appropriate Trust in AI through Transparency and Explainability
Kim, S. S. Y. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3638184 · Cited by 28
“In contrast, users with unwarranted trust (trusting when the AI is untrustworthy) and unwarranted distrust (distrusting when the AI is trustworthy) are likely to have low-quality interactions, even unsafe ones when they are using the AI in high-stakes settings [1, 17, 36, 49].”
McGovern, A., Demuth, J. L., Bostrom, A., Wirz, C. D., Tissot, P. E., Cains, M. G., & Musgrave, K. D. (2024) · npj natural hazards. · doi:10.1038/s44304-024-00014-x · Cited by 21
Chen, C., Liao, M., & Sundar, S. S. (2024) · International Symposium on Trustworthy Autonomous Systems (TAS) · doi:10.1145/3686038.3686066 · Cited by 29
“One major goal of explanations is to help users develop appropriate levels of trust in machine learning models [26].”
Pirzado, F. A., Ahmed, A., Alipour, M. A., & Terashima‐Marín, H. (2025) · Journal of Computers in Education · doi:10.1007/s40692-025-00375-w · Cited by 23
“Over-trust may lead students to accept AI-generated outputs without proper evaluation, resulting in the integration of incorrect or suboptimal code into their assignments (Jacovi et al., 2021).”
Controlling Machine Translation for Multiple Attributes with Additive Interventions
Schioppa, A., Vilar, D., Sokolov, A., & Filippova, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.535 · Cited by 39
“This in turns increases the MT system’s trustworthiness by providing an explicit contract (Jacovi et al., 2021), formulated as “whenever there is an ambiguity, we enable users’ agency”.”
BackgroundMitigating Algorithm Aversion in Recruiting: A Study on Explainable AI for Conversational Agents
Fleiß, J., Bäck, E., & Thalmann, S. (2024) · ACM SIGMIS Database the DATABASE for Advances in Information Systems · doi:10.1145/3645057.3645062 · Cited by 25
“On the one hand, an understanding of the inner workings can be formed from repeated interactions with the system and may even lead to blind trust or automation bias (Jacovi et al., 2021).”
BackgroundStriking the Balance in Using LLMs for Fact-Checking: A Narrative Literature Review
Dierickx, L., van Dalen, A., Opdahl, A. L., & Lindén, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71210-4_1 · Cited by 26
Kahr, P. K., Rooks, G., Snijders, C. C. P., & Willemsen, M. C. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713099 · Cited by 26
Supporting Co-Adaptive Machine Teaching through Human Concept Learning and Cognitive Theories
Gebreegziabher, S. A., Yang, Y., Glassman, E. L., & Li, T. J.-J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713708 · Cited by 22
Bonsai: Intentional and Personalized Social Media Feeds
Malki, O. E., Quéré, M. A. L., Monroy‐Hernández, A., & Ribeiro, M. H. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791855 · Cited by 24
Structured Verification of Machine Learning Models in Industrial Settings
Kaminwar, S. R., Goschenhofer, J., Thomas, J., Thon, I., & Bischl, B. (2022) · Big Data · doi:10.1089/big.2021.0112 · Cited by 23
“d test cases in this phase that need to be implemented in later phases to verify the model’s prediction quality and ensure trust in the model’s correctness.(38) Another important aspect of this phase is the definition of baseline methods jointly with the Business Experts.”
MethodToward a taxonomy of trust for probabilistic machine learning
Broderick, T., Gelman, A., Meager, R., Smith, A. L., & Zheng, T. (2023) · Science Advances · doi:10.1126/sciadv.abn3999 · Cited by 25
A Quantum Model of Trust Calibration in Human–AI Interactions
Roeder, L., Hoyte, P., van der Meer, J. N., Fell, L., Johnston, P., Kerr, G., & Bruza, P. (2023) · Entropy · doi:10.3390/e25091362 · Cited by 19
How to Evaluate Explainability? - A Case for Three Criteria
Speith, T. (2022) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew56159.2022.00024 · Cited by 21
“Justified trust is based on trustworthiness [23], [24].”
Background
Metadata
- Authors
- Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y.
- Affiliations
- Bar-Ilan University (IL)
- University of Washington (US)
- The University of Melbourne (AU)
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 624-635
- OpenAlex
- W3134111219
- Cited by
- 545
- References
- 74
- Citations by year
- 2026: 131 · 2025: 137 · 2024: 102 · 2023: 90 · 2022: 60 · 2021: 24
- FWCI
- 31.88
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- Funders
- European Research Council (802774)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- trust in AI, interpersonal trust, contractual trust, trustworthiness, user vulnerability, explainable AI, human-AI interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.