Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
The DOI matches its Crossref record; 74 of the 76 references it cites were found in a registry or an index, and 2 are not checked yet.
Abstract
Trust is a central component of the interaction between people and AI, in that 'incorrect' levels of trust may cause misuse, abuse or disuse of the technology. But what, precisely, is the nature of trust in AI? What are the prerequisites and goals of the cognitive mechanism of trust, and how can we promote them, or assess whether they are being satisfied in a given interaction? This work aims to answer these questions. We discuss a model of trust inspired by, but not identical to, interpersonal trust (i.e., trust between people) as defined by sociologists. This model rests on two key properties: the vulnerability of the user; and the ability to anticipate the impact of the AI model's decisions. We incorporate a formalization of 'contractual trust', such that trust between a user and an AI model is trust that some implicit or explicit contract will hold, and a formalization of 'trustworthiness' (that detaches from the notion of trustworthiness in sociology), and with it concepts of 'warranted' and 'unwarranted' trust. We present the possible causes of warranted trust as intrinsic reasoning and extrinsic behavior, and discuss how to design trustworthy AI, how to evaluate whether trust has manifested, and whether it is warranted. Finally, we elucidate the connection between trust and XAI using our formalization.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Uncertainty is a defining characteristic of trust (Mayer et al., 1995) and has been referred to as a necessary prerequisite of human trust in AI that has been lacking in current XAI studies (Jacovi et al., 2021).”
From Exploring the effects of human-centered AI explanations on trust and reliance (2023) · cites this paper as Jacovi et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“research found that people often ignore recommendations because they do not trust them”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The primary motivation for clinical XAI is fostering appropriate trust: enabling clinicians to calibrate their reliance on AI recommendations based on the quality of supporting reasoning [156,157].”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [157]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, people find it difficult to correctly calibrate their trust in a decision aid [10,13,21,37].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“For example, [38] formalizes human-AI trust by considering and extending the concepts of interpersonal trust.”
From Trustworthy AI: From Principles to Practices (2023) · cites this paper as [38]
9 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736 · cited 1×
“on basic definition of trust regards it as a directional transaction between two parties: if A believes that B will act in A’s best interest, and accepts vulnerability to B’s actions, then A trusts B [49]. The goal of trust is to “make social life predictable [by anticipating the impact of behavior], and make it easie…”
BackgroundTrust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 2×
“ay that the trust is warranted if it is the result of trustworthiness, and otherwise it is unwarranted [50]. Warranted trust is sometimes referred to as trust which is calibrated with trustworthiness [43]. In other words, trust is the cognitive mechanism to give the ‘illusion’ of anticipating intended behavior, which…”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 2×
“s can reveal strong correlations between attributes for one of those classes, giving leads to research on causation between them, even if the model was not useful for the original classification task [42, 46]. 2Assume that the performance evaluation is representative of real usage for now, although this is an importan…”
BackgroundInfluentialDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“Transparency High-standard documentation All Technical explainability Factsheets (explainability) Saliency maps [61], self-attention patterns [40], influence functions [38], probing [16] Adaptable user-centered explainability Factsheets (explainability) Counterfactual [21], contrastive [51], natural language [27], by-…”
BackgroundExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Self-citationHumans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554 · cited 2×
“8We consider AI as an automation attributed with intent, and thus, Human-AI trust is a sub-case of human-machine trust [28, 55].”
BackgroundLewis, J. D., & Weigert, A. (1985) · Social Forces · doi:10.2307/2578601 · Cited by 4,163 · cited 1×
“the influence of the trustee’s incentives on Human-AI trust, should build on the existing research in sociology that investigates the influence of relationships and communities on interpersonal trust [44]. Consequently, the question of whether this trust is warranted or unwarranted is ill-defined.Our definition of war…”
BackgroundA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“We set the “effort boundary” around 774 tokens—a few of minutes of reading—and the “vulnerability boundary” right from natural language inference [7], commonsense QA [70], and multi-sentence”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Baier, A. (1986) · Ethics · doi:10.1086/292745 · Cited by 2,567 · cited 1×
“present two directions for extension of our formalization inspired by other factors of interpersonal trust. 9.1 Other Elements of Interpersonal Trust and Trust in Automation Trust vs Reliance. Baier [4] introduces the term reliance for trust in inanimate objects, such as trusting a stool to hold your weight. We may fe…”
BackgroundInfluential
Research works printed but not linked: 27
- [1] David Alvarez Melis and Tommi Jaakkola. 2018. Towards Robust Interpretability with Self-Explaining Neural Networks. In Advances in Neural Information Processing Systems 31 , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett (Eds.). 7775-7784. http://papers.nips.cc/paper/8003towards-robust-interpretability-with-self-explaining-neural-networks.pdf -
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- [15] Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 (2017). - · arXiv
- 10.18653/v1 · doi:10.18653/v1
- [21] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1412.6572 - · arXiv
- [22] Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Counterfactual Visual Explanations (Proceedings of Machine Learning Research, Vol. 97) , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, Long Beach, California, USA, 2376-2384. http://proceedings.mlr.press/v97/goyal19a.html -
- 10.18653/v1 · doi:10.18653/v1
- [25] Sven Ove Hansson. 2018. Risk. In The Stanford Encyclopedia of Philosophy (fall 2018 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University. -
- [29] Robert R Hoffman. 2017. A taxonomy of emergent trusting in the human-machine relationship. Cognitive systems engineering: The future for a changing world (2017), 137-163. -
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- [40] Hana Kopecka and Jose M Such. 2020. Explainable AI for Cultural Minds. https://sites.google.com/view/dexahai-at-ecai2020/home Workshop on Dialogue, Explanation and Argumentation for Human-Agent Interaction , DEXAHAI ; Conference date: 07-09-2020. -
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- [53] Carolyn McLeod. 2015. Trust. In The Stanford Encyclopedia of Philosophy (fall 2015 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University. -
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- [64] Philipp Schmidt and Felix Bießmann. 2019. Quantifying Interpretability and Trust in Machine Learning Systems. CoRR abs/1901.08558 (2019). arXiv:1901.08558 http://arxiv.org/abs/1901.08558 - · arXiv
- [67] Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. 2020. We Need to Talk About Random Splits. CoRR abs/2005.00636 (2020). arXiv:2005.00636 https://arxiv.org/abs/2005.00636 - · arXiv
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
Cited by
Items 701–747 of 747
BISHT, H. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7506138
Bunea, O.-I., Sabie, O. M., Popescu, R. I., Corboș, R.-A., & Nastacă, C. C. (2026) · Administrative Sciences · doi:10.3390/admsci16100463
Barati, F., Soltanian, M. R. K., Borna, K., & Longo, L. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100304
Sethi, S., & Singh, S. (2026) · ACM Transactions on Social Computing · doi:10.1145/3849483
Lifecycle Narrative: Reconstructing the Development Process of a Machine Learning Model
Gröhn, T., Fritzsche, M.-C., Snell, K., Marttinen, P., & Ylikoski, P. (2026) · ACM Journal on Responsible Computing · doi:10.1145/3850237
“For instance, ethical and legal aspects, in addition to the model’s internal properties, are important motivations for improving model transparency and trustworthiness [13, 20, 29, 33].”
Assessing The Impact of Transparent AI Systems in Enhancing User Trust and Privacy
Bodimani, M. (2024) · Journal of Science & Technology · Cited by 7
Witczak, O., Dewalska–Opitek, A., Walusiak, Ł., Szostak, A., & Kalisz, J. (2026) · EUROPEAN RESEARCH STUDIES JOURNAL · doi:10.35808/ersj/4434 · Cited by 1
Martin, P. P., Rost, M., Koenen, J., & Graulich, N. (2026) · Science & Education · doi:10.1007/s11191-026-00789-7
Artificial Intelligence as a Catalyst for Green Economy and Sustainable Development
Amalia, L., & Astuty, P. (2025) · Cognitionis Civitatis et Politicae
Rojas, E. E. S., Moreno, G. A. B., & Aguirre, J. L. E. (2026) · World Symposium on Artificial Intelligence (WSAI) · doi:10.1109/wsai70435.2026.11708676
“Target: (cid:305) 85% [14].”
Tang, G. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2739765
Trustworthy AI Is Not Valuable
Lim, J. (2026) · Journal of Applied Philosophy · doi:10.1002/japp.70137
Foundations of Explainable and Responsible AI in Cybersecurity
Khan, I. A., & Samdani, F. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-2600-1970-2.ch001
Tilannetietoisuuden tukeminen ihmisen ja tekoälyn tiimeissä läpinäkyvyyden ja selitettävyyden avulla
Mallat, H. (2026) · Aaltodoc (Aalto University)
Ganesh, P., & Namdeo, A. K. (2026) · Journal of Advances in Science and Technology · doi:10.29070/w0j61s95
Chuang, Y.-H., Riswanto, D., & Halim, S. A. B. E. (2026) · Journal of Human Interaction and Social Studies · doi:10.65310/8v0m4m50
“The pattern is consistent with the relational and institutional literature (Choung et al., 2023; Jacovi et al., 2021).”
Chen, T.-C. T. (2026) · Cognitive Computation · doi:10.1007/s12559-026-10666-8
“…In addition, it evaluates trustworthiness utilizing existing performance metrics, including estimation accuracy, inter-example distance, and sample size, thereby avoiding reliance on multiple external evaluation tools, such as fairness checklists, factsheets, reproducibility check-lists, etc. [20].”
Yamamoto, K., Tanikawa, T., & Kuzuoka, H. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-02469-5
“Existing frameworks for interpersonal, human-automation, human-AI, and network trust focus on different relationships and settings [8, 16–19].”
Using AI, Doubting AI: Developer Mental Models and Trust in AI-Assisted Software Work
DSILVA, N. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2743445
Trust Hinders Bidirectional Human-AI AlignmentPreprint
Benk, M., & Sokol, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.15136
“In this context, trust in an automated system is considered appropriate, calibrated or warranted when it is caused by the trustworthiness of the system [9].”
InfluentialUnderstanding Human-AI Trust in Military AI Applications: A Multistage Human Factors Framework
Kim, K. (2026) · International journal of industrial engineering
Liu, S., Song, Y., Li, H., Wang, Z., Lan, M., & Lv, F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.21233
“Trust can be influenced by system performance, user experience, and interaction context, and may change during continuous use [7].”
Karlsson, M., Kuai, J., Brantner, C., Yu, H., & Meese, J. (2026) · Journal of Broadcasting & Electronic Media · doi:10.1080/08838151.2026.2751043
Grimaldi, C., D’Arco, L., Rossi, A., & Rossi, S. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7649037
Ahmed, T., Rahman, S. A., Rocky, K. H., Sukrana, B., Tabassum, T., & Hasan, M. J. (2026) · SAGE Open · doi:10.1177/21582440261500232
Djanggih, H. (2026) · Research Square · doi:10.21203/rs.3.rs-11460998/v1
Who Is Sycophantic to Whom? Confident LLMs, Human Deference, and the Collapse of Trust CalibrationPreprint
Deepthika, S., Yadav, P., & Sankar, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.31579
“The design goal is not more trust but calibrated trust: trust that matches a system’s actual capability (Lee and See, 2004; Wischnewski et al., 2023), or, in the terms of Jacovi et al. (2021), warranted rather than unwarranted trust.”
Public Trust in AI Governance and Support for Algorithmic Accountability in Bangladesh
Faysal, M. A. T., Islam, S., Mafuz, M., & Abdul, M. K. (2026) · Journal of Policies and Recommendations · doi:10.64907/xkmf.v5i4.jopr.3
The Impact of AI-Driven Information Technology on Social Media Communication and Consumer Trust
Rahman, M. H., Afsar, T. B., & Alam, M. S. (2026) · doi:10.63125/2jq2bx46
Khanna, P., & Ramin, K. (2026) · European Safety and Reliability Conference (ESREL) · doi:10.3850/esrel2026061419_esrel26-p26521-cd
Garg, A. (2023) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.5557
“Recommendations from AI will only be accepted based on their reliability, confidence, and validation by humans (Jacovi et al. 2021 Therefore, the recommendation acceptance rate is 88%.”
Procedural Reliability Safeguards for AI Assisted Digital Evidence Analysis in Bangladesh
Anny, M. D. I., Saha, A., Islam, M. A., & Mannan, K. A. (2026) · Theoretical and applied technological science review · doi:10.64907/xkmf.v4i4.tatscr.2
Explanation and Validation as Conditions for Trust in Judicial AI in Bangladesh
Ferdous, M., Maisha, I. J., Maisha, I. J., & Mannan, K. A. (2026) · Journal of State Government and Mass Media · doi:10.64907/xkmf.v04i04.jsgmm.1
Climate Justice and Conditions for Trustworthy AI Risk Information in Bangladesh
Mridha, M. E., Akter, S., Islam, T., & Mannan, K. A. (2026) · Journal of Pandemic Disaster and Recession · doi:10.64907/xkmf.v6.i4.jpdr.1
Islam, M. D., Oaishe, S. S., Joy, M. H., & Mannan, K. A. (2026) · Journal of Gender Education and Society · doi:10.64907/xkmf.v5.i4.jges.1
When do people trust AI in healthcare? The role of risk and oversight communication
Wan, Y. (2026) · Journal of Psychology and AI · doi:10.1080/29974100.2026.2757444
Ikejima, S., Ohara, R., Koseki, Y., Narumi, T., & Kuzuoka, H. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3831338
Explanations That Work: A Risk-Based Model for Meaningful AI Transparency in Regulated Decisions
Shepherd, N. J. (2025) · Journal of AI-powered medical innovations. · doi:10.60087/japmi.vol.04.issue.01.id.04
Gawade, Y. (2024) · The Eastasouth Journal of Information System and Computer Science · doi:10.58812/esiscs.v2i01.1307
Pessianzadeh, A., Sultana, N., den Bulck, H. V., Gefen, D., Jabbari, S., & Rezapour, R. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.x.823
Pick Your Poison: Effects of Miscalibrated AI Advice and Silent Updates on Reliance
Spillner, L., Rockstroh, J., Wenig, N., Drescher, L., Ringe, R., Zargham, N., Porzel, R., & Malaka, R. (2026) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3841580.3841606
Huang, K., & Wang, Y. (2026) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3841580.3845756
Are we measuring trust correctly in explainability, interpretability, and transparency research?
Miller, T. (2022) · arXiv · doi:10.48550/arxiv.2209.00651
“[14] build on this model by first acknowledging that human-machine trust is similar to directional, interpersonal trust, except where the trustee is a machine doing some action, such as providing advice, finding information, executing a command, etc.”
BackgroundSelf-citationDistrust in (X)AI -- Measurement Artifact or Distinct Construct?
Scharowski, N., & Perrig, S. A. C. (2023) · arXiv · doi:10.48550/arxiv.2303.16495
“[22], warranted trust [16], or reliance [33]).”
Background
Metadata
- Authors
- Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y.
- Affiliations
- Bar-Ilan University (IL)
- University of Washington (US)
- The University of Melbourne (AU)
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 624-635
- OpenAlex
- W3134111219
- Cited by
- 545
- References
- 74
- Citations by year
- 2026: 131 · 2025: 137 · 2024: 102 · 2023: 90 · 2022: 60 · 2021: 24
- FWCI
- 31.88
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- Funders
- European Research Council (802774)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- trust in AI, interpersonal trust, contractual trust, trustworthiness, user vulnerability, explainable AI, human-AI interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.