The DOI matches its Crossref record; 101 of the 109 references it cites were found in a registry or an index; 24 of the 96 sources its claims cite were checked for support.
Explainable recommendation: when design meets trust calibration
Open accessAbstract
Human-AI collaborative decision-making tools are being increasingly applied in critical domains such as healthcare. However, these tools are often seen as closed and intransparent for human decision-makers. An essential requirement for their success is the ability to provide explanations about themselves that are understandable and meaningful to the users. While explanations generally have positive connotations, studies showed that the assumption behind users interacting and engaging with these explanations could introduce trust calibration errors such as facilitating irrational or less thoughtful agreement or disagreement with the AI recommendation. In this paper, we explore how to help trust calibration through explanation interaction design. Our research method included two main phases. We first conducted a think-aloud study with 16 participants aiming to reveal main trust calibration errors concerning explainability in AI-Human collaborative decision-making tools. Then, we conducted two co-design sessions with eight participants to identify design principles and techniques for explanations that help trust calibration. As a conclusion of our research, we provide five design principles: Design for engagement, challenging habitual actions, attention guidance, friction and support training and learning. Our findings are meant to pave the way towards a more integrated framework for designing explanations with trust calibration as a primary goal.
Overview
CoChat Summary
## TL;DR
This qualitative study examines how explanation interface design can support trust calibration in Human-AI collaborative decision-making, using a screening-prescription mock-up with medical experts. From a think-aloud phase and two co-design sessions, the authors derive five design principles for XAI: engagement, challenging habitual actions, attention guidance, friction, and training/learning.
## Key findings
- In the exploration phase, many participants made trust calibration errors by skipping or misapplying explanations; skipping was grouped into five categories: lack of curiosity, perceived goal impediment, redundant information, perceived complexity, and lack of context.
- The study recruited 24 participants total: 16 in the exploration phase and 8 in the co-design phase.
- In the design phase, participants proposed four main Human-Explanation interaction themes: abstraction, cues, control, and adaptation.
- The final contribution is five design principles for calibrated-trust XAI: design for engagement, challenging habitual actions, attention guidance, friction, and support training and learning.
## Methods
- Two-phase qualitative design: (1) a think-aloud exploration phase and (2) two co-design sessions to derive interface principles and techniques.
- Sample and setting: medical experts from oncology departments in three UK organisations; the use case was an AI-supported screening prescription task in a web-based mock-up.
- Analysis: audio transcriptions, observations, notes, and sketches were analyzed using content analysis with iterative coding and verification by the coauthors.
## Limitations
- The paper states that the results are qualitative and that future controlled experiments are needed to map trust calibration errors to explanation types and to test whether the proposed design techniques have the intended effects.
- The paper notes it was unable to pair configurations of the XAI model with the type of bias and error; this limits how specific the design guidance can be.
- Evident from the text: the study used a mock-up and a single domain-specific scenario (screening prescriptions), so generalization beyond this context is not established in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures







Verification
Claims the paper makes
19 Valid · 0 Not valid · 55 Not checked · 129 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“They found that communicating explanations, on average, is not improving trust calibration, i.e., users still end-up in situations where they over-trust or under-trust AI-based recommendation.”
- Also stated in: § Theoretical background and related work · plain
“Recent studies showed that even though explanations were communicated to people, trust calibration is not improved [16, 96].”
- Also stated in: § Theoretical background and related work · plain
“Overall, users of XAI systems fail, on average, to calibrate their trust, i.e., human decision-maker working collaboratively with an AI can still be notably following incorrect recommendations or rejecting correct ones.”
§ Explainable recommendation: when design meets trust calibration > Introduction
What the sources say
- Supportedcites [7] Bansal, G., Wu, T., Zhu, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M.T., Weld, D.S.: Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. arXiv preprint arXiv:2006.14779 (2020) source
The Introduction (p. 2) reports no significant overall improvement from explanations over showing AI confidence alone, despite gains when the AI was correct and losses when it erred. Section 6.2 (p. 12) says explanations increased blind trust rather than appropriate reliance, supporting the claim that explanations did not reliably calibrate users’ trust. (LLM, conf=0.87)
- Partly supportedcites [16] Bussone, A., Stumpf, S. & O'sullivan, D.: The role of explanations on trust and reliance in clinical decision support systems, pp. 160–169. IEEE (2015) sourceExperimental — accuracy not yet measured
The Conclusion (p. 11) reports that comprehensive explanations promoted over-reliance, while selective explanations promoted self-reliance, supporting the claim that users can still over- or under-rely on recommendations. It does not report an average effect showing that communicating explanations fails to improve trust calibration; the Discussion (p. 11) instead says explanations can help users assess suggestions and that their effects require further study. (LLM, conf=0.86)
- Partly supportedcites [96] Zhang, Y., Liao, Q.V., Bellamy, R.K.E.: Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. Proceedings of the 2020 conference on fairness, accountability, and transparency. Association for Computing Machinery, Barcelona (2020) sourceExperimental — accuracy not yet measured
The Results (§4.2) report that local explanations did not improve trust calibration, and §2 describes calibration as distinguishing when to trust or distrust AI. The claim broadens this finding from the particular local explanations tested to communicating explanations generally; the paper (§5.3) allows that other explanation methods may be more effective. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Theoretical background and related work · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Another research [49] showed that current explainable models are designed to be used by ML engineers for debugging purposes where end-users are not the main target of these explanations.”
§ Explainable recommendation: when design meets trust calibration > Introduction
What the sources say
- Partly supportedcites [49] Miller, T., Howe, P., Sonenberg, L.: Explainable AI: beware of inmates running the asylum or: how I learnt to stop worrying and love the social and behavioural sciences. arXiv preprint arXiv:1712.00547 (2017) sourceExperimental — accuracy not yet measured
The Abstract (p. 1) argues that most AI researchers are building explanatory agents for themselves rather than their intended users, supporting the point that end-users are often not the main target. The paper does not say that current explainable models are designed for ML engineers to use for debugging; 'explanatory debugging' appears only in the title of a paper listed in the appendix (p. 6). (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, this introduces a lack of transparency which can weaken the users’ ability to calibrate trust in the Human-AI collaborative decision-making environments.”
- Also stated in: § Introduction · plain
“This means the human decision-maker is less able to judge when to follow a system output or reject it [33].”
§ Explainable recommendation: when design meets trust calibration > Introduction
What the sources say
- Partly supportedcites [33] Gui, C., Chan, V.: Machine learning in medicine. Unive. West. Ont. Med. J. 86, 76–78 (2017) sourceExperimental — accuracy not yet measured
The 'challenges to translation' section says that black-box ML models can make the medical community uncomfortable because their mechanisms are unknown. This supports a concern about transparency and trust, but it does not address users’ ability to calibrate trust in human–AI collaborative decision-making environments. Page information is unavailable in our copy. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although, many studies showed that explanation could indeed improve trust calibration [80].”
§ Explainable recommendation: when design meets trust calibration > Theoretical background and related work
What the sources say
- Partly supportedcites [80] Tomsett, R., Preece, A., Braines, D., Cerutti, F., Chakraborty, S., Srivastava, M., Pearson, G., Kaplan, L.: Rapid trust calibration through interpretable and uncertainty-aware AI. Patterns 1, 100049 (2020) sourceExperimental — accuracy not yet measured
The Results section proposes that explanations, alongside uncertainty estimates, could help people calibrate trust in AI. But the Discussion section says research on explanations’ effects remains to be done and notes that explanations can sometimes make users overly trusting. The paper does not report that many studies have shown explanations improve trust calibration. (LLM, conf=0.87)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recently, trust calibration became an essential metric for evaluating collaborative Human-AI decision-making tools [7, 16, 96].”
§ Explainable recommendation: when design meets trust calibration > Introduction
What the sources say
- Partly supportedcites [7] Bansal, G., Wu, T., Zhu, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M.T., Weld, D.S.: Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. arXiv preprint arXiv:2006.14779 (2020) sourceExperimental — accuracy not yet measured
Section 6.2 (p. 12) treats appropriate reliance rather than blind trust as important to human–AI collaboration. But Section 2 (p. 3) says the paper evaluates these tools using end-to-end team accuracy; it does not establish trust calibration as an essential evaluation metric. (LLM, conf=0.82)
- Partly supportedcites [16] Bussone, A., Stumpf, S. & O'sullivan, D.: The role of explanations on trust and reliance in clinical decision support systems, pp. 160–169. IEEE (2015) sourceExperimental — accuracy not yet measured
The paper treats appropriately balanced trust and reliance as important in clinical decision support (Assessing System Reliability and Trust, p. 4; Discussion, p. 11), and measures participants’ trust and reliance in its study (Data Collection and Analysis, p. 6). It does not establish trust calibration as an essential evaluation metric for collaborative human–AI tools generally. (LLM, conf=0.86)
- Supportedcites [96] Zhang, Y., Liao, Q.V., Bellamy, R.K.E.: Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. Proceedings of the 2020 conference on fairness, accountability, and transparency. Association for Computing Machinery, Barcelona (2020) source
The Introduction (page unavailable) calls prediction-specific trust calibration especially important to the success of AI-assisted decision-making. Section 5.3 (page unavailable) says its effect should be evaluated with users in the loop and presents the study as an example of that evaluation, supporting its role as an important evaluation measure. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
89 more claims · 4 with their cited sources checked
Create a free account to see all 203 claims and every source verdict.
- Probably not a substantial claim. 82
- It may state two separate claims as one. 49
- The quote only partly states this claim. 1
Citations
Cites
(2011) · Cited by 14,317
(1999) · Cited by 7,995
An introduction to cybernetics
Ashby, W. R. (1956) · doi:10.5962/bhl.title.5851 · Cited by 7,578
Affect and Cognition Based Trust as Foundations for Interpersonal Cooperation in Organizations
McAllister, D. J. (1995) · Academy of Management Journal · doi:10.2307/256727 · Cited by 7,346
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 4×
“Previous research [40, 88] identified five primary contexts where trust calibration errors in automation occur, their reasons of occurrences and potential design solutions.”
BackgroundResultInfluentialA Simple Model of Herd Behavior
Banerjee, A. V. (1992) · The Quarterly Journal of Economics · doi:10.2307/2118364 · Cited by 6,556
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Designing the user interface: Strategies for effective human-computer interaction
(1993) · Applied Ergonomics · doi:10.1016/0003-6870(93)90471-k · Cited by 5,587
Explanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302 · cited 2×
“Some users prefer a detailed explanation, while others need a brief explanation based on their curiosity level [47].”
Background
Research works printed but not linked: 2
- Biran, O., Cotton, C: Explanation and justification in machine learning: a survey, pp. 8–13 (2017)
- Bradley, S.: Design Fundamentals: Elements, Attributes, & Principles. Vanseo design, Colorado (2013)
Cited by
Rosenbacke, R., Melhus, Å., McKee, M., & Stuckler, D. (2024) · JMIR AI · doi:10.2196/53207 · Cited by 305
“Naiseh et al [28,29] noted that explanations affected both cognitive and affect-based trust and could result in either overtrust or undertrust.”
BackgroundMethodInfluentialNaiseh, M., Al-Thani, D., Jiang, N., & Ali, R. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102941 · Cited by 170
Self-citationMehrotra, S., Degachi, C., Vereschak, O., Jonker, C. M., & Tielman, M. L. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3696449 · Cited by 254
“The most common is agreement percentage , that is the percentage of trials in Another measure of appropriate trust is related to switch percentage [178], A final method is to measure ideal trusting behavior during the task beforehand and compare to which extent the”
InfluentialBeyond AI-powered context-aware services: the role of human–AI collaboration
Jiang, N., Liu, X., Liu, H., Lim, E. T. K., Tan, C., & Gu, J. (2022) · Industrial Management & Data Systems · doi:10.1108/imds-03-2022-0152 · Cited by 157
Self-citationRezaeian, O., Bayrak, A. E., & Asan, O. (2026) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2025.2539458 · Cited by 39
“Explainable Artificial Intelligence (XAI) has become essential in fields like medicine[42, 31], finance, and autonomous systems, where understanding AI decisions is critical.”
Metadata
- Authors
- Naiseh, M., Al-Thani, D., Jiang, N., & Ali, R.
- Affiliations
- Bournemouth University (GB)
- Hamad bin Khalifa University (QA)
- Corresponding author
- Naiseh, Mohammad
- Jiang, Nan
- Ali, Raian
- Venue
- World Wide Web
- Published
- 2021-08-02
- Volume
- 24
- Issue
- 5
- Pages
- 1857-1884
- ISSN
- 1386-145X (ISSN-L)
- 1573-1413
- PMC
- PMC8327305
- OpenAlex
- W3191257699
- Semantic Scholar
- e632f1191b3c261cdae48698c97eb2e9a5a45c78
- Cited by
- 68
- References
- 109
- Citations by year
- 2026: 20 · 2025: 20 · 2024: 17 · 2023: 3 · 2022: 7 · 2021: 1
- FWCI
- 2.46
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- World Wide Web (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Bournemouth University Research Online (Bournemouth University) (acceptedVersion)
- ePrints Soton (University of Southampton) (submittedVersion, cc-by)
- Aberdeen University Research Archive (Aberdeen University) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Funders
- iQ HealthTech (01101)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- explainable recommendation, trust calibration, AI transparency, attention guidance, clinical decision support, think-aloud study, co-design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Explainable recommendation: when design meets trust calibration. https://app.cochat.ai/papers/10.1007/s11280-021-00916-0
- The paper
- Naiseh, M., Al-Thani, D., Jiang, N., & Ali, R. (2021). Explainable recommendation: when design meets trust calibration. World Wide Web. https://doi.org/10.1007/s11280-021-00916-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.