Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
The DOI matches its Crossref record; 74 of the 76 references it cites were found in a registry or an index, and 2 are not checked yet.
Abstract
Trust is a central component of the interaction between people and AI, in that 'incorrect' levels of trust may cause misuse, abuse or disuse of the technology. But what, precisely, is the nature of trust in AI? What are the prerequisites and goals of the cognitive mechanism of trust, and how can we promote them, or assess whether they are being satisfied in a given interaction? This work aims to answer these questions. We discuss a model of trust inspired by, but not identical to, interpersonal trust (i.e., trust between people) as defined by sociologists. This model rests on two key properties: the vulnerability of the user; and the ability to anticipate the impact of the AI model's decisions. We incorporate a formalization of 'contractual trust', such that trust between a user and an AI model is trust that some implicit or explicit contract will hold, and a formalization of 'trustworthiness' (that detaches from the notion of trustworthiness in sociology), and with it concepts of 'warranted' and 'unwarranted' trust. We present the possible causes of warranted trust as intrinsic reasoning and extrinsic behavior, and discuss how to design trustworthy AI, how to evaluate whether trust has manifested, and whether it is warranted. Finally, we elucidate the connection between trust and XAI using our formalization.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Uncertainty is a defining characteristic of trust (Mayer et al., 1995) and has been referred to as a necessary prerequisite of human trust in AI that has been lacking in current XAI studies (Jacovi et al., 2021).”
From Exploring the effects of human-centered AI explanations on trust and reliance (2023) · cites this paper as Jacovi et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“research found that people often ignore recommendations because they do not trust them”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The primary motivation for clinical XAI is fostering appropriate trust: enabling clinicians to calibrate their reliance on AI recommendations based on the quality of supporting reasoning [156,157].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [157]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, people find it difficult to correctly calibrate their trust in a decision aid [10,13,21,37].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“For example, [38] formalizes human-AI trust by considering and extending the concepts of interpersonal trust.”
From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [38]
9 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736 · cited 1×
“on basic definition of trust regards it as a directional transaction between two parties: if A believes that B will act in A’s best interest, and accepts vulnerability to B’s actions, then A trusts B [49]. The goal of trust is to “make social life predictable [by anticipating the impact of behavior], and make it easie…”
BackgroundTrust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 2×
“ay that the trust is warranted if it is the result of trustworthiness, and otherwise it is unwarranted [50]. Warranted trust is sometimes referred to as trust which is calibrated with trustworthiness [43]. In other words, trust is the cognitive mechanism to give the ‘illusion’ of anticipating intended behavior, which…”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 2×
“s can reveal strong correlations between attributes for one of those classes, giving leads to research on causation between them, even if the model was not useful for the original classification task [42, 46]. 2Assume that the performance evaluation is representative of real usage for now, although this is an importan…”
BackgroundInfluentialDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“Transparency High-standard documentation All Technical explainability Factsheets (explainability) Saliency maps [61], self-attention patterns [40], influence functions [38], probing [16] Adaptable user-centered explainability Factsheets (explainability) Counterfactual [21], contrastive [51], natural language [27], by-…”
BackgroundExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Self-citationHumans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 2×
“8We consider AI as an automation attributed with intent, and thus, Human-AI trust is a sub-case of human-machine trust [28, 55].”
BackgroundLewis, J. D., & Weigert, A. (1985) · Social Forces · doi:10.2307/2578601 · Cited by 4,163 · cited 1×
“the influence of the trustee’s incentives on Human-AI trust, should build on the existing research in sociology that investigates the influence of relationships and communities on interpersonal trust [44]. Consequently, the question of whether this trust is warranted or unwarranted is ill-defined.Our definition of war…”
BackgroundA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“We set the “effort boundary” around 774 tokens—a few of minutes of reading—and the “vulnerability boundary” right from natural language inference [7], commonsense QA [70], and multi-sentence”
BackgroundCommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1421 · Cited by 2,829 · cited 1×
“We set the “effort boundary” around 774 tokens—a few of minutes of reading—and the “vulnerability boundary” right from natural language inference [7], commonsense QA [70], and multi-sentence”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Research works printed but not linked: 27
- David Alvarez Melis and Tommi Jaakkola. 2018. Towards Robust Interpretability with Self-Explaining Neural Networks. In Advances in Neural Information Processing Systems 31 S. Bengio H. Wallach H. Larochelle K. Grauman N. Cesa-Bianchi and R. Garnett (Eds.). 7775--7784. http://papers.nips.cc/paper/8003-towards-robust-interpretability-with-self-explaining-neural-networks.pdf
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 (2017). · arXiv
- 10.18653/v1 · doi:10.18653/v1
- Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1412.6572 · arXiv
- Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Counterfactual Visual Explanations (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, Long Beach, California, USA, 2376--2384. http://proceedings.mlr.press/v97/goyal19a.html
- 10.18653/v1 · doi:10.18653/v1
- Sven Ove Hansson. 2018. Risk. In The Stanford Encyclopedia of Philosophy (fall 2018 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.
- Robert R Hoffman. 2017. A taxonomy of emergent trusting in the human-machine relationship. Cognitive systems engineering: The future for a changing world (2017), 137--163.
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- Hana Kopecka and Jose M Such. 2020. Explainable AI for Cultural Minds. https://sites.google.com/view/dexahai-at-ecai2020/home Workshop on Dialogue, Explanation and Argumentation for Human-Agent Interaction, DEXAHAI; Conference date: 07-09-2020.
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- Carolyn McLeod. 2015. Trust. In The Stanford Encyclopedia of Philosophy (fall 2015 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- Philipp Schmidt and Felix Bießmann. 2019. Quantifying Interpretability and Trust in Machine Learning Systems. CoRR abs/1901.08558 (2019). arXiv:1901.08558 http://arxiv.org/abs/1901.08558 · arXiv
- Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. 2020. We Need to Talk About Random Splits. CoRR abs/2005.00636 (2020). arXiv:2005.00636 https://arxiv.org/abs/2005.00636 · arXiv
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Algorithmic fairness in artificial intelligence for medicine and healthcare
Chen, R. J., Wang, J. J., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Lu, M. Y., Sahai, S., & Mahmood, F. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01056-8 · Cited by 750
“Such contracts are AI functionalities that are anticipated to have known vulnerabilities 305 .”
BackgroundTrustworthy AI: From Principles to Practices
Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., & Zhou, B. (2023) · ACM Computing Surveys · doi:10.1145/3555803 · Cited by 612
Trust in AI: progress, challenges, and future directions
Afroogh, S., Akbari, A., Malone, E., Kargar, M., & Alambeigi, H. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-04044-8 · Cited by 507
“In Jacovi et al. (2021b), a precise discussion is presented regarding the nature of trust in AI, as well as the prerequisites and goals of the cognitive mechanism of trust.”
InfluentialUnderspecification Presents Challenges for Credibility in Modern Machine Learning
D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. (2020) · arXiv · doi:10.48550/arxiv.2011.03395 · Cited by 398
“Strategies of this type are one way to warrant trust in an ML system (Jacovi et al., 2020).”
BackgroundA Systematic Literature Review of User Trust in AI-Enabled Systems: An HCI Perspective
Bach, T. A., Khan, A., Hallock, H., Beltrão, G., & Sousa, S. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2138826 · Cited by 377
“A growing number of researchers argue that fostering and maintaining user trust is the key to calibrating the user-AI relationship (Jacovi et al., 2021; Shin, 2021), achieving trust-worthy AI (Smith, 2019), and further unlocking the potential of AI for society (Bughin et al., 2018; Cheatham et al.,…”
BackgroundHarnessing the power of AI in qualitative research: Exploring, using and redesigning ChatGPT
Zhang, H., Wu, C., Xie, J., Lyu, Y., Cai, J., & Carroll, J. M. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100144 · Cited by 393
“Trust is crucial in human-AI collaboration, with perceived risk, AI capabilities, expectations, and user vulnerability shaping the degree of trust users place in AI [49].”
How transparency modulates trust in artificial intelligence
Zerilli, J., Bhatt, U., & Weller, A. (2022) · Patterns · doi:10.1016/j.patter.2022.100455 · Cited by 257
Responsible AI Pattern Catalogue: A Collection of Best Practices for AI Governance and Engineering
Lu, Q., Zhu, L., Xu, X., Whittle, J., Zowghi, D., & Jacquet, A. (2023) · ACM Computing Surveys · doi:10.1145/3626234 · Cited by 270
“To try to fill the principle-algorithmic gap, further guidance such as guidebooks, 2 , 3 questions to generate discussions [66, 67], checklists [44, 62], and documentation templates [1, 4, 52, 55, 94, 127] have started to appear.”
BackgroundCausal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
Feder, A., Keith, K. A., Manzoor, E., Pryzant, R., Sridhar, D., Wood-Doughty, Z., Eisenstein, J., Grimmer, J., Reichart, R., Roberts, M. E., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00511 · Cited by 220
“Yet correlational predictive models can be untrustworthy (Jacovi et al., 2021): They may latch onto spurious correlations (“shortcuts”), leading to errors in out-of-distribution (OOD) settings (e.”
BackgroundMehrotra, S., Degachi, C., Vereschak, O., Jonker, C. M., & Tielman, M. L. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3696449 · Cited by 254
“; Our set of future research directions results from a structured summary of our analysis based on the implications of the approaches (definitions, methods, tasks and measures) adopted by the authors to foster appropriate trust in human-AI interaction.”
InfluentialGood proctor or “big brother”? Ethics of online exam supervision technologies
Coghlan, S., Miller, T., & Paterson, J. (2021) · Philosophy & Technology · doi:10.1007/s13347-021-00476-1 · Cited by 219
“Values such as liberty and trust in technology (Jacovi et al., 2021) and in educational institutions are important for democratic society more generally.”
BackgroundSelf-citationAppropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584066 · Cited by 208
Lai, V., Chen, C., Smith-Renner, A., Liao, Q. V., & Tan, C. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594087 · Cited by 204
Miller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172
Self-citationLukyanenko, R., Maass, W., & Storey, V. C. (2022) · Electronic Markets · doi:10.1007/s12525-022-00605-4 · Cited by 171
“The academic literature developed a number of general theoretical models, focusing on the nomological network of trust in AI; that is, antecedents of trust in AI and its consequences (Asan et al., 2020; Jacovi et al., 2021; Lansing & Sunyaev, 2016; Siau & Wang, 2018; Söllner & Leimeister, 2013).”
Explainable AI Over the Internet of Things (IoT): Overview, State-of-the-Art and Future Directions
Jagatheesaperumal, S. K., Pham, Q.-V., Ruby, R., Yang, Z., Xu, C., & Zhang, Z. (2022) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2022.3215676 · Cited by 171
“Significant challenges imposed by the AI systems due to their opacity of black-box models often give the threat to trust from the ethical perspective [14].”
Designing for Responsible Trust in AI Systems: A Communication Perspective
Liao, Q. V., & Sundar, S. S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533182 · Cited by 165
How Explainability Contributes to Trust in AI
Ferrario, A., & Loi, M. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533202 · Cited by 163
Riedl, R. (2022) · Electronic Markets · doi:10.1007/s12525-022-00594-4 · Cited by 181
“In another recent paper, Jacovi et al. (2021) make an explicit call for personality research in the context of trust in AI systems—they write: “Personal Attributes of the Trustor [.”
BackgroundMethodInfluentialWischnewski, M., Krämer, N., & Müller, E. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581197 · Cited by 146
“However, unlike Lee and See [74], Jacovi and colleagues [53] have argued that calibrated trust "is trust that is caused by trustworthiness (to some contract)" (p.”
BackgroundInfluentialTrust in Generative AI among students: An Exploratory Study
Amoozadeh, M., Daniels, D., Nam, D., Kumar, A., Chen, S., Hilton, M., Ragavan, S. S., & Alipour, M. A. (2023) · arXiv · doi:10.48550/arxiv.2310.04631 · Cited by 143
“At this juncture, one factor, namely students’ trust in GenAI, becomes paramount in whether and how students use GenAI in classrooms (and whether and how instructors should adopt these tools in classrooms).”
BackgroundExplainable Artificial Intelligence (XAI) in Insurance
Owens, E., Sheehan, B., Mullins, M., Cunneen, M., Ressel, J., & Castignani, G. (2022) · Risks · doi:10.3390/risks10120230 · Cited by 181
“The recent proliferation of XAI literature is partly driven by the need to maintain users’ trust in AI to further develop AI adoption (Jacovi et al. 2021; Robinson 2020).”
BackgroundZhang, R., Duan, W., Flathmann, C., McNeese, N., Freeman, G., & Williams, A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610072 · Cited by 134
Park, K., & Yoon, H. Y. (2024) · Public Relations Review · doi:10.1016/j.pubrev.2024.102507 · Cited by 145
Supporting Artificial Social Intelligence With Theory of Mind
Williams, J., Fiore, S. M., & Jentsch, F. G. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.750763 · Cited by 128
CRAFT: Concept Recursive Activation FacTorization for Explainability
Fel, T., Picard, A. M., Béthune, L., Boissin, T., Vigouroux, D., Colin, J., Cadénc, R., & Serre, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00266 · Cited by 249
Reframing Human-AI Collaboration for Generating Free-Text Explanations
Wiegreffe, S., Hessel, J., Swayamdipta, S., Riedl, M. O., & Choi, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.47 · Cited by 202
How do we assess the trustworthiness of AI? Introducing the trustworthiness assessment model (TrAM)
Schlicker, N., Baum, K. L., Uhde, A., Sterz, S., Hirsch, M. C., & Langer, M. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108671 · Cited by 122
Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making
Schoeffer, J., De‐Arteaga, M., & Kühl, N. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642621 · Cited by 164
“A series of prior studies have found that knowledge about the features that an AI model uses influences people’s fairness perceptions [49–51, 88, 98, 122].”
BackgroundEvaluating pointwise reliability of machine learning prediction
Nicora, G., Rios, M., Abu–Hanna, A., & Bellazzi, R. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.103996 · Cited by 116
González-Gonzalo, C., Thee, E. F., Klaver, C. C. W., Lee, A. Y., Schlingemann, R. O., Tufail, A., Verbraak, F., & Sánchez, C. I. (2022) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2021.101034 · Cited by 103
“3 claims, are urged to ensure beneficial societal outcomes from AI (Jacovi et al., 2021).”
Benk, M., Kerstan, S., von Wangenheim, F., & Ferrario, A. (2024) · AI & Society · doi:10.1007/s00146-024-02059-y · Cited by 120
“…dynamics (e.g., goals, success criteria, trustworthiness criteria of the AI(Salimzadeh et al. 2023), socio-technical context (Ehsan and Riedl 2020; Benk et al. 2022), and finally (d) the risk assessment (e.g., interaction risks (Jacovi et al. 2021), possibly aligned with the EU AI Act a .”
InfluentialOpen Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101
“In this way, it could be possible to gain confidence that models will (or will not) behave in a safe way without necessarily conducting extensive testing of the models [Jacovi et al., 2021].”
BackgroundInteractive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101
Artificial Intelligence Across Europe: A Study on Awareness, Attitude and Trust
Scantamburlo, T., Cortés, À., Foffano, F., Barruè, C., Distefano, V., Pham, L., & Fabris, A. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3461633 · Cited by 120
“AI and trust recall a vast academic literature investigating shared principles among ethics guidelines [33, 29, 36], as well as challenges and future directions [61, 52, 37, 30, 49].”
Is Everyone an Artist? A Study on User Experience of AI-Based Painting System
Xu, J., Zhang, X., Li, H., Yoo, C., & Pan, Y. (2023) · Applied Sciences · doi:10.3390/app13116496 · Cited by 107
Will AI Console Me when I Lose my Pet? Understanding Perceptions of AI-Mediated Email Writing
Liu, Y., Mittal, A., Yang, D., & Bruckman, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517731 · Cited by 92
“This validates the theory on intrinsic trust [27] that says such trust can only be gained when the user has background knowledge on what behavior is trustworthy for a given task the AI performs.”
BackgroundTowards a roadmap on software engineering for responsible AI
Lu, Q., Zhu, L., Xu, X., Whittle, J., & Xing, Z. (2022) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3522664.3528607 · Cited by 91
”Because AI is 100% right and safe”: User Attitudes and Sources of AI Authority in India
Kapania, S., Siy, O., Clapper, G., SP, A. M., & Sambasivan, N. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517533 · Cited by 90
Building Trustworthy AI Solutions: A Case for Practical Solutions for Small Businesses
Crockett, K. A., Colyer, E., Gerber, L., & Latham, A. (2021) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2021.3137091 · Cited by 88
“the benefits of public involvement to build trust in AI tech [50], [51], SMEs indicated that they were not sure how to involve the public and that the real benefits of consulting with the”
BackgroundThe effects of explanations on automation bias
Vered, M., Livni, T., Howe, P. D. L., Miller, T., & Sonenberg, L. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.103952 · Cited by 111
Self-citationInvestigating and Designing for Trust in AI-powered Code Generation Tools
Wang, R., Cheng, R., Ford, D., & Zimmermann, T. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658984 · Cited by 87
“Previous research highlighted the importance of context in understanding users’ trust in AI systems [35, 39, 67].”
BackgroundInfluentialInterpretable Machine Learning for Discovery: Statistical Challenges and Opportunities
Allen, G. I., Gan, L., & Zheng, L. (2023) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-040120-030919 · Cited by 115
Waldman, A. E., & Martin, K. E. (2022) · Big Data & Society · doi:10.1177/20539517221100449 · Cited by 95
Bostrom, A., Demuth, J. L., Wirz, C. D., Cains, M. G., Schumacher, A., Madlambayan, D., Bansal, A. S., Bearth, A., Chase, R. J., Crosman, K. M., et al. (2023) · Risk Analysis · doi:10.1111/risa.14245 · Cited by 90
“Although trust can be described as warranted or unwarranted (Jacovi et al., 2021), whether trust can actually be calibrated to an external standard that defines trustworthiness is debated.”
BackgroundInfluentialA Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
Suresh, H., & Guttag, J. (2021) · Equity and Access in Algorithms, Mechanisms, and Optimization · doi:10.1145/3465416.3483305 · Cited by 82
“Mitigating deployment bias might involve systems that help users balance their faith in model predictions with other information and judgements [28].”
BackgroundArtificial intelligence in veterinary diagnostic imaging: Perspectives and limitations
Burti, S., Banzato, T., Coghlan, S., Wodziński, M., Bendazzoli, M., & Zotti, A. (2024) · Research in Veterinary Science · doi:10.1016/j.rvsc.2024.105317 · Cited by 95
Fietta, V., Zecchinato, F., Stasi, B. D., Polato, M., & Monaro, M. (2022) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2021.3125280 · Cited by 81
Making It Possible for the Auditing of AI: A Systematic Review of AI Audits and AI Auditability
Li, Y., & Goel, S. (2024) · Information Systems Frontiers · doi:10.1007/s10796-024-10508-8 · Cited by 102
How Do We Assess the Trustworthiness of AI? Introducing the Trustworthiness Assessment Model (TrAM)
Schlicker, N., Baum, K., Uhde, A., Sterz, S., Hirsch, M. C., & Langer, M. (2022) · doi:10.31234/osf.io/qhwvx · Cited by 79
1 + 1 > 2? Information, Humans, and Machines
Lu, T., & Zhang, Y. (2024) · Information Systems Research · doi:10.1287/isre.2023.0305 · Cited by 119
Singh, G., Moncrieff, G. R., Venter, Z. S., Cawse-Nicholson, K. A., Slingsby, J. A., & Robinson, T. B. (2024) · Scientific Reports · doi:10.1038/s41598-024-65954-w · Cited by 104
“Overall, 243 datasets were assessed.”
Trust criteria for artificial intelligence in health: normative and epistemic considerations
Kostick, K. M., Lang, B. H., Smith, J. N., Hurley, M. E., & Blumenthal‐Barby, J. (2023) · Journal of Medical Ethics · doi:10.1136/jme-2023-109338 · Cited by 75
“8 28–31 As such, there is not likely to be a ‘one size fits all’ level at which trust in AI systems should be considered ‘appropriately’ calibrated.”
BackgroundExplainable machine learning for public policy: Use cases, gaps, and research directions
Amarasinghe, K., Rodolfa, K. T., Lamba, H., & Ghani, R. (2023) · Data & Policy · doi:10.1017/dap.2023.2 · Cited by 73
Local Interpretations for Explainable Natural Language Processing: A Survey
Luo, S., Ivison, H., Han, S. C., & Poon, J. (2024) · ACM Computing Surveys · doi:10.1145/3649450 · Cited by 103
Large language models present new questions for decision support
Handler, A., Larsen, K. R., & Hackathorn, R. D. (2024) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2024.102811 · Cited by 69
Capable but Amoral? Comparing AI and Human Expert Collaboration in Ethical Decision Making
Tolmeijer, S., Christen, M., Kandul, S., Kneer, M., & Bernstein, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517732 · Cited by 70
Pasricha, S. (2022) · IEEE Consumer Electronics Magazine · doi:10.1109/mce.2022.3220001 · Cited by 86
“, because of undertrust).(10) AI can be abused when implemented without due regard for the medical practitioners’ capabilities or the patients’ interests.”
BackgroundWang, F., Li, N., Cheung, A. C. K., & Wong, G. K. W. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-025-00547-9 · Cited by 69
“Conversely, the appropriateness of AI use can be compromised by disuse, misuse, and abuse (Jacovi et al., 2021).”
InfluentialKiyasseh, D., Laca, J., Haque, T. F., Miles, B. J., Wagner, C., Donoho, D. A., Anandkumar, A., & Hung, A. J. (2023) · Communications Medicine · doi:10.1038/s43856-023-00263-3 · Cited by 69
“Consistent explanations can, for example, improve the trustworthiness of AI systems 37 .”
BackgroundMcNeese, N. J., Flathmann, C., O’Neill, T., & Salas, E. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.107874 · Cited by 73
Lyu, W., Zhang, S., Chung, T., Sun, Y., & Zhang, Y. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100383 · Cited by 68
“For example, Jacovi et al. [55] define trust and distrust in AI as a human perceiving an AI model as trustworthy and accepting vulnerability to the model’s actions or not.”
Schoeffer, J., Kuehl, N., & Machowski, Y. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533218 · Cited by 67
Artificial intelligence in cancer: applications, challenges, and future perspectives
Cheng, H., & Shi, S.-S. (2025) · Molecular Cancer · doi:10.1186/s12943-025-02450-3 · Cited by 76
“In parallel, explainability is critical for trust, so developing interpretable methods that help clinicians understand model behavior and diagnose why disparities arise is essential for regulatory approval and for sustainable clinical adoption [287, 288].”
Misplaced Trust and Distrust: How Not to Engage with Medical Artificial Intelligence
Starke, G., & Ienca, M. (2022) · Cambridge Quarterly of Healthcare Ethics · doi:10.1017/s0963180122000445 · Cited by 65
“Journal of Medical Ethics 2020;46(7):478–81.”
Ferrario, A., Facchini, A., & Termine, A. (2024) · Minds and Machines · doi:10.1007/s11023-024-09681-1 · Cited by 64
“For instance, an AI can be 31 According to this account, trustworthiness follows from a commitment, namely, the commitment of maintaining selected functionalities that are deemed of relevance for an application (Jacovi et al., 2021).”
Harm to Nonhuman Animals from AI: a Systematic Account and Framework
Coghlan, S., & Parker, C. (2023) · Philosophy & Technology · doi:10.1007/s13347-023-00627-6 · Cited by 75
“4 Note that the terms abuse, misuse, and disuse can have technical and various meanings in AI and technology literature (Jacovi et al., 2021; Parasuraman & Riley, 1997).”
BackgroundCertification Labels for Trustworthy AI: Insights From an Empirical Mixed-Method Study
Scharowski, N., Benk, M., Kühne, S. J., Wettstein, L., & Brühlmann, F. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3593994 · Cited by 74
“2The definition of trust in AI and its operationalization is an ongoing debate [31, 56, 65, 67].”
BackgroundUnderstanding the Evolvement of Trust Over Time within Human-AI Teams
Duan, W., Zhou, S., Scalia, M. J., Yin, X., Weng, N., Zhang, R., Freeman, G., McNeese, N. J., Gorman, J. C., & Tolston, M. T. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3687060 · Cited by 78
Starke, G., Gille, F., Termine, A., Aquino, Y. S. J., Chavarriaga, R., Ferrario, A., Hastings, J., Jongsma, K. R., Kellmeyer, P., Kulynych, B., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/56306 · Cited by 67
“The combination of fragmented conceptual research, practical concerns, and implementation difficulties inhibits fostering warranted trust in AI (ie, trust caused by the trustworthiness of the AI) [25,28], which will be increasingly crucial for delivering health care [29].”
On the Relation of Trust and Explainability: Why to Engineer for Trustworthiness
Kastner, L., Langer, M., Lazar, V., Schomacker, A., Speith, T., & Sterz, S. (2021) · 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew53955.2021.00031 · Cited by 61
Explanation-Based Human Debugging of NLP Models: A Survey
Lertvittayakumjorn, P., & Toni, F. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00440 · Cited by 90
“Some researchers have explored various merits of explanations to humans, such as supporting human decision makings (Lai and Tan, 2019), increasing human trust in AI (Jacovi et al., 2020) and even teaching humans to perform challenging tasks (Lai et al.”
BackgroundWhat kind of trust does AI deserve, if any?
Alvarado, R. (2023) · AI and Ethics · doi:10.1007/s43681-022-00224-x · Cited by 60
Đerić, E., Frank, D., & Milković, M. (2025) · Information · doi:10.3390/info16070622 · Cited by 60
Metzger, L., Miller, L., Baumann, M., & Kraus, J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642122 · Cited by 72
Seven HCI Grand Challenges Revisited: Five-Year Progress
Stephanidis, C., Salvendy, G., Antona, M., Duffy, V. G., Gao, Q., Karwowski, W., Konomi, S., Nah, F. F., Ntoa, S., Rau, P. P., et al. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2450411 · Cited by 69
“Building and maintaining an appropriate level of trust between humans and AI systems is fundamental to ensuring sustainable human-AI teaming interactions (Jacovi et al., 2021).”
Lee, C., & Cha, K. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102976 · Cited by 71
Ulmer, D., Hardmeier, C., & Frellsen, J. (2021) · IT University Of Copenhagen (IT University of Copenhagen) · doi:10.48550/arxiv.2110.03051 · Cited by 69
“Other Approaches to Uncertainty Quantification The need for the quantification of uncertainty in order to earn the trust of end-users and stakeholders has been a key driver for research (Bhatt et al., 2021; Jacovi et al., 2021; Liao & Sundar, 2022).”
BackgroundVisser, R., Peters, T. M., Scharlau, I., & Hammer, B. (2025) · Cognitive Systems Research · doi:10.1016/j.cogsys.2025.101357 · Cited by 58
El–Sappagh, S., Alonso, J. M., Abuhmed, T., Ali, F., & Bugarín, A. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10415-5 · Cited by 68
Hauptman, A. I., Schelble, B. G., Duan, W., Flathmann, C., & McNeese, N. J. (2024) · Cognition Technology & Work · doi:10.1007/s10111-024-00765-7 · Cited by 67
“This is because it allows them to predict the AI’s future behavior (Jacovi et al. 2021).”
The Art of Saying No: Contextual Noncompliance in Language Models
Brahman, F., Kumar, S., Balachandran, V., Dasigi, P., Pyatkin, V., Ravichander, A., Wiegreffe, S., Dziri, N., Chandu, K., Hessel, J., et al. (2024) · neural information processing systems · doi:10.52202/079017-1573 · Cited by 157
Exploring the effects of human-centered AI explanations on trust and reliance
Scharowski, N., Perrig, S. A. C., Svab, M., Opwis, K., & Brühlmann, F. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1151150 · Cited by 57
‘Are You OK?’ Students’ Trust in a Chatbot Providing Support Opportunities
Pesonen, J. A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77943-6_13 · Cited by 56
“Both too high and low levels of trust may cause misuse, abuse, or disuse of the technology [20].”
BackgroundA Dempster-Shafer Approach to Trustworthy AI With Application to Fetal Brain MRI Segmentation
Fidon, L., Aertsen, M., Kofler, F., Bink, A., David, A. L., Deprest, T., Emam, D., Guffens, F., András, J., Kasprian, G. J., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3346330 · Cited by 67
“An AI algorithm is defined as trustworthy with respect to a contract of trust if it provides guarantees that it will abide by the obligations of said contract [24].”
BackgroundLanger, M., Hunsicker, T., Feldkamp, T., König, C. J., & Grgić-Hlača, N. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517527 · Cited by 55
“Specifically, varying the term used to refer to ADM systems may affect whether people positively or negatively evaluate the use of said system in a respective context, and may lead to a lack of acceptance or disuse just due to terminological differences and not actual differences in system-design or functionalities [3…”
BackgroundEthics of using artificial intelligence (AI) in veterinary medicine
Coghlan, S., & Quinn, T. P. (2023) · AI & Society · doi:10.1007/s00146-023-01686-1 · Cited by 63
“Unwar-ranted trust in AI can cause its misuse, while unwar-ranted distrust can cause disuse that deprives patients of benefits (Jacovi et al. 2021).”
Perceptions About Generative AI and ChatGPT Use by Faculty and College Students
Petricini, T., Wu, C., & Zipf, S. T. (2023) · doi:10.35542/osf.io/jyma4 · Cited by 54
Duenser, A., & Douglas, D. (2023) · IEEE Intelligent Systems · doi:10.1109/mis.2023.3322586 · Cited by 54
A Machine Learning Explainability Tutorial for Atmospheric Sciences
Flora, M. L., Potvin, C. K., McGovern, A., & Handler, S. L. (2023) · Artificial Intelligence for the Earth Systems · doi:10.1175/aies-d-23-0018.1 · Cited by 76
Toward the dynamic relationship between AI transparency and trust in AI: A case study on ChatGPT
Lee, C., & Cha, K. (2025) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2405266 · Cited by 53
“The finding that AI systems with higher performance and job relevance would be more trustworthy not only supports the moderating effect of functionality and helpfulness but also supports their direct impact on the overall trust in AI (Jacovi et al., 2021).”
Kim, M., Kim, S., Kim, J., Song, T.-J., & Kim, Y. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2023.103160 · Cited by 53
AI Trust: Can Explainable AI Enhance Warranted Trust?
de Brito Duarte, R., Correia, F., Arriaga, P., & Paiva, A. (2023) · Human Behavior and Emerging Technologies · doi:10.1155/2023/4637678 · Cited by 52
Artificial agents’ explainability to support trust: considerations on timing and context
Papagni, G., de Pagter, J., Zafari, S., Filzmoser, M., & Koeszegi, S. T. (2023) · AI & SOCIETY · doi:10.1007/s00146-022-01462-7 · Cited by 52
“Accordingly, Jacovi et al. (2021) argue that an AI model (i.e., the ‘ locus ’ of the decision-making processes) is trust-worthy if it acts consistently according to specific ‘contracts’ the model or artificial agent entertains with a user.”
BackgroundMethodInfluentialExplainable software systems: from requirements analysis to system evaluation
Chazette, L., Brunotte, W., & Speith, T. (2022) · Requirements Engineering · doi:10.1007/s00766-022-00393-5 · Cited by 61
“Although one could trust an untrustworthy system, this trust would be unjustified and inadequate [118].”
BackgroundA data-centric approach for ethical and trustworthy AI in journalism
Dierickx, L., Opdahl, A. L., Khan, S. A., Lindén, C., & Rojas, D. C. G. (2024) · Ethics and Information Technology · doi:10.1007/s10676-024-09801-6 · Cited by 56
“In machine learning, however, the paradox of trusting the results of a machine learning system is that trustworthiness is not a prerequisite for trust: “Trust can exist in a model that is not trustworthy, and a trustworthy model does not necessarily gain trust” (Jacovi et al., 2021, p. 627).”
McGrath, M. J., Duenser, A., Lacey, J., & Paris, C. (2025) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2025.100200 · Cited by 51
Influences on User Trust in Healthcare Artificial Intelligence: A Systematic Review
Jermutus, E., Kneale, D., Thomas, J., & Michie, S. (2022) · Wellcome Open Research · doi:10.12688/wellcomeopenres.17550.1 · Cited by 51
“The majority of the papers (18) were excluded after reading the full text as their research design did not match the required criterion, i.”
BackgroundVereschak, O., Alizadeh, F., Bailly, G., & Caramiaux, B. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642018 · Cited by 72
Metadata
- Authors
- Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y.
- Affiliations
- Bar-Ilan University (IL)
- University of Washington (US)
- The University of Melbourne (AU)
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 624-635
- OpenAlex
- W3134111219
- Cited by
- 545
- References
- 74
- Citations by year
- 2026: 131 · 2025: 137 · 2024: 102 · 2023: 90 · 2022: 60 · 2021: 24
- FWCI
- 31.88
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- Funders
- European Research Council (802774)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- trust in AI, interpersonal trust, contractual trust, trustworthiness, user vulnerability, explainable AI, human-AI interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.