The DOI matches its Crossref record; 51 of the 55 references it cites were found in a registry or an index, and 4 are not checked yet; 20 of the 65 sources its claims cite were checked for support.
Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust
Open accessAbstract
Trustworthy Artificial Intelligence (AI) is characterized, among other things, by: 1) competence, 2) transparency, and 3) fairness. However, end-users may fail to recognize incompetent AI, allowing untrustworthy AI to exaggerate its competence under the guise of transparency to gain unfair advantage over other trustworthy AI. Here, we conducted an experiment with 120 participants to test if untrustworthy AI can deceive end-users to gain their trust. Participants interacted with two AI-based chess engines, trustworthy (competent, fair) and untrustworthy (incompetent, unfair), that coached participants by suggesting chess moves in three games against another engine opponent. We varied coaches' transparency about their competence (with the untrustworthy one always exaggerating its competence). We quantified and objectively measured participants' trust based on how often participants relied on coaches' move recommendations. Participants showed inability to assess AI competence by misplacing their trust with the untrustworthy AI, confirming its ability to deceive. Our work calls for design of interactions to help end-users assess AI trustworthiness.
Verified 2026-10-06 · 4 references not checked yet
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
7 Valid · 2 Not valid · 11 Not checked · 49 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“It did not even need to generate misleading explanations [30,46].”
§ Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust > 4 DISCUSSION
What the sources say
- Disputedcites [30] sourceExperimental — accuracy not yet measured
Our copy of Lakkaraju & Bastani (2020) says the opposite of what the citing sentence attributes: in the user study (Sec. 5.2 / 'User study design', results paragraph) only 9.1% of domain experts who saw the untrustworthy black box itself trusted it, and only 10% of those shown E2 (which exposes the prohibited features race and gender), whereas trust rose to 70% and 88% only when participants were shown the deliberately misleading explanations E1 and E3 — an increase the Introduction (p. 1, 'Contributions') quantifies as 9.8 times. That is, in this paper the misleading explanations are precisely what was needed to manufacture trust in an untrustworthy model; the framework in Sec. 3 likewise defines a 'potentially misleading' explanation as the mechanism (O(E) ≠ O*(B)) by which trust is inflated. No passage in our copy supports the assertion that an untrustworthy system 'did not even need to generate misleading explanations'; the paper's construction stage (Sec. 5.1) explicitly builds such explanations in order to obtain trust. (LLM, conf=0.65)
Checked by a reasoning model — may be wrong.
- Not checked yetcites [46]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Unfortunately, existing methods that deliver AI explanations [10,21,43] might not be enough to counter such deception.”
§ Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust > 1 INTRODUCTION
What the sources say
- Not checked yetcites [10] source
- Partly supportedcites [21] sourceExperimental — accuracy not yet measured
Our copy does support the weaker half of the attribution: the Abstract (p. 1) states 'we discuss why current approaches to explanatory methods especially for deep neural networks are insufficient', and the Conclusions (pp. 8-9) assert that satisfactory explanation remains out of reach and that current work is 'siloed'. The paper also touches deception in two places — the Introduction (p. 1) on adversarial examples and trojaning attacks making DNNs 'easily fooled', and Interpretability vs. Completeness (p. 2), where the authors call explanations 'inherently misleading' when optimized to hide undesirable attributes. However, the source never claims that explanation methods fail specifically as a defence against deception by untrustworthy AI: its stated insufficiency concerns transparency, standardization and systematic evaluation, and its deception discussion runs partly in the opposite direction (explanations themselves deceiving users). The 'counter such deception' qualifier is therefore the citing authors' extension, not this paper's point. (LLM, conf=0.72)
- Not checked yetcites [43]
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, most existing explainable AI (XAI) methods [10,21,43] target math-savy AI creators [1,22] to aid them in AI model debugging and monitoring [6].”
§ Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust > 2 RELATED WORK
What the sources say
- Not checked yetcites [10] source
- Could not verifycites [21] sourceExperimental — accuracy not yet measured
The citation [1,22] is attached to the description of XAI methods as targeting “math-savy AI creators.” The Introduction (pp. 1–2) says this survey aims to engage the machine-learning community, but it does not establish who most existing XAI methods target. Searches for mathematical expertise, developers, and intended users found no passage addressing that attribution. (LLM, conf=0.72)
- Not checked yetcites [43]
- Not checked yetcites [1] source
- Partly supportedcites [22] sourceExperimental — accuracy not yet measured
The citation to this paper is attached to the description of AI creators as math-savvy, not to the separate claim about debugging and monitoring. Section 2.2 (Internalist; page unavailable) describes computer scientists’ algorithmic evaluations as centered on mathematically defined measures, which supports the math-focused characterization. It does not establish that existing XAI methods target those creators. (LLM, conf=0.75)
- Supportedcites [6] source
Citation [6] is attached to the claim that explanations aid model debugging and monitoring. Section 3.1, “Explainability Needs” (page unavailable), describes explainability as a debugging solution and discusses using explanations to assess how feature drift affects model outcomes during monitoring. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“naive application of existing XAI methods may be ineective [2,40,65] to broader audience of end-users, if not harmful [28,47,49,51].”
§ Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust > 4 DISCUSSION
What the sources say
- Not checked yetcites [2] source
- Supportedcites [40] source
The Findings (§3.2) report that lay participants shown a clear model were less able to detect and correct its sizable mistakes. The Discussion and Conclusion (§8) likewise warns that exposing model internals by default may be detrimental. This supports the cited possibility of harm from applying an interpretability approach to end-users, though the experiments concerned laypeople using linear regression for real-estate valuation. (LLM, conf=0.88)
- Supportedcites [65] source
The Results (section 4.2) report that local explanations did not improve trust calibration or AI-assisted predictions. The Related Work section says that information overload can potentially worsen users’ situation awareness and decision outcomes. The claim is hedged as a possibility, not a claim that this study demonstrated harm. (LLM, conf=0.86)
- Not checked yetcites [28] source
- Partly supportedcites [47] sourceExperimental — accuracy not yet measured
The Ripple Effect Trap section (3.4; page unavailable) says technologies introduced without accounting for social context may undermine social goals, exacerbate problems, and introduce new ones. That supports the possibility of harm, the part of the sentence to which this citation is attached. The paper discusses fair-ML systems rather than the naive application of existing XAI methods to a broad audience of end-users, so it does not establish that more specific claim. (LLM, conf=0.83)
- Not checked yetcites [49] source
- Not checked yetcites [51]
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, in the 14CA0~0; condition, the two coaches were able to make signicantly fewer conversions (? = 0.0100) between the rst game (&1 = 0, <4380= = 1, &3 = 6) and the third game (&1 = 0, <4380= = 0, &3 = 2).”
§ Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust > 3.5.4 Participant Conversion.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
29 more claims
Create a free account to see all 69 claims and to verify the 45 references still open.
- It may state two separate claims as one. 14
- Probably not a substantial claim. 27
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust > 3.1 Method
“2) D=CADBCF>AC⌘~-the coach had lower competence than the opponent and the other coach, and it misrepresented its competence to the participant.”
§ Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust > 5 CONCLUSION AND FUTURE WORK
“| betrayal | [2325, 2375] [2325, 2375] [2425, 2475] | [2425, 2475] | [2425, 2475] | [2525, 2575] |”
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Affect and Cognition Based Trust as Foundations for Interpersonal Cooperation in Organizations
McAllister, D. J. (1995) · Academy of Management Journal · doi:10.2307/256727 · Cited by 7,346
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Algorithm aversion: People erroneously avoid algorithms after seeing them err.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015) · Journal of Experimental Psychology: General · doi:10.1037/xge0000033 · Cited by 3,232 · cited 1×
“Also, the drop in concurrence in the later games shows diminishing reliance on both coaches, since by then most participants already experienced negative outcomes (i.e., game losses) [11].”
BackgroundThe aligned rank transform for nonparametric factorial analyses using only anova procedures
Wobbrock, J. O., Findlater, L., Gergle, D., & Higgins, J. J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978963 · Cited by 2,854
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 2×
“However, most existing explainable AI (XAI) methods [10, 21, 43] target math-savy AI creators [1, 22] to aid them in AI model debugging and monitoring [6].”
Background
Research works printed but not linked: 12
- Arpad E. Elo. 1967. The Proposed USCF Rating System, Its Development, Theory, and Applications. Chess Life, Vol. 22, 8 (August 1967), 242--247.
- Birhanu Eshete. 2021. Making machine learning trustworthy. Science, Vol. 373, 6556 (2021), 743--744.
- 10.1111/j.1751--9004.2009.00232.x · doi:10.1111/j.1751--9004.2009.00232.x
- Lex Fridman, Li Ding, Benedikt Jenik, and Bryan Reimer. 2019. Arguing Machines: Human Supervision of Black Box AI Systems That Make Life-Critical Decisions. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16--20, 2019. Computer Vision Foundation / IEEE, 146--154. http://openaccess.thecvf.com/content_CVPRW_2019/html/Autonomous_Driving/Fridman_Arguing_Machines_Human_Supervision_of_Black_Box_AI_Systems_That_CVPRW_2019_paper.html
- Stéphane Hulaud. 2018. Identification of taste attributes from an audio signal. US Patent 9,934,785.
- 10.1038/s42256-019-0088--2 · doi:10.1038/s42256-019-0088--2
- Niklas Luhmann. 1988. Familiarity, Confidence, Trust: Problems and Alternatives. D. Gambetta, editor, Trust: Making and Breaking of Cooperative Relations, Basil Blackwell, Oxford, 1988 (1988).
- Maria Madsen and Shirley Gregor. 2000. Measuring human-computer trust. In Proceedings of the 11 th Australasian Conference on Information Systems. 6--8.
- Johannes Schneider, Joshua Handali, Michalis Vlachos, and Christian Meske. 2020. Deceptive AI Explanations: Creation and Detection. CoRR, Vol. abs/2001.07641 (2020). showeprint[arXiv]2001.07641 https://arxiv.org/abs/2001.07641 · arXiv
- Simone Stumpf, Adrian Bussone, and Dympna O'sullivan. 2016. Explanations considered harmful? user interactions with machine learning systems. In The CHI 2016 Human Centred Machine Learning Workshop.
- Jennifer Wortman Vaughan and Hanna Wallach. 2020. A human-centered agenda for intelligible machine learning. Machines We Trust: Getting Along with Artificial Intelligence (2020).
- 10.1145/3025171.3025219 · DOI · Paper page
Cited by
AI deception: A survey of examples, risks, and potential solutions
Park, P. S., Goldstein, S., O’Gara, A., Chen, M., & Hendrycks, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100988 · Cited by 243
“Although the deceptive reasoning is transparent through the chain of thought, the malicious behavior remained even when the chain of thought was ‘‘distilled’’ away and not made visible to users.”
AI Deception: A Survey of Examples, Risks, and Potential Solutions
Park, P. S., Goldstein, S., O'Gara, A., Chen, M., & Hendrycks, D. (2023) · arXiv · doi:10.48550/arxiv.2308.14752 · Cited by 243
“Although the deceptive reasoning is transparent through the chain of thought, the malicious behavior remained even when the chain of thought was ‘‘distilled’’ away and not made visible to users.”
Jabbour, S., Fouhey, D., Shepard, S., Valley, T. S., Kazerooni, E. A., Banovic, N., Wiens, J., & Sjoding, M. W. (2023) · JAMA · doi:10.1001/jama.2023.22295 · Cited by 236
Prabhudesai, S., Yang, L., Asthana, S., Huan, X., Liao, Q. V., & Banovic, N. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584033 · Cited by 82
“Most existing quantitative empirical studies in ML-supported decision-making measure the effects of transparency and communicating uncertainty on efficiency [4, 102], quality of decisions [36, 69, 73], trust and reliance [5, 7, 18, 19, 77, 108], and the decision-maker’s confidence in decisions [51].”
BackgroundExplanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making
Schoeffer, J., De‐Arteaga, M., & Kühl, N. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642621 · Cited by 164
“Importantly, even if explanations lead to more (in)accurate decisions, it is unclear from prior work how this relates to distributive fairness properties.”
Background
Metadata
- Authors
- Banovic, N., Yang, Z., Ramesh, A., & Liu, A.
- Affiliations
- University of Michigan, Ann Arbor, MI, USA
- University of Michigan (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2023-04-14
- Volume
- 7
- Issue
- CSCW1
- Pages
- 1-17
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W4365999041
- Semantic Scholar
- 4c2857dbcef303ef8451d272f2f2688e07e013a0
- Cited by
- 55
- References
- 51
- Citations by year
- 2026: 13 · 2025: 21 · 2024: 16 · 2023: 4
- FWCI
- 13.54
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion, other-oa)
- Funders
- University of Michigan
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- trustworthy AI, AI deception, end-user trust, transparency, fairness, chess engines, human-AI interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust. https://app.cochat.ai/papers/10.1145/3579460
- The paper
- Banovic, N., Yang, Z., Ramesh, A., & Liu, A. (2023). Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3579460
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.