CoChat

The DOI matches its Crossref record; 83 of the 83 references it cites were found in a registry or an index; 23 of the 39 sources its claims cite were checked for support.

Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations

Open access
Schemmer, Max, Kuehl, Niklas, Benz, Carina, Bartos, Andrea, Satzger, Gerhard · 2023 · Proceedings of the 28th International Conference on Intelligent User Interfaces

Abstract

AI advice is becoming increasingly popular, e.g., in investment and medical treatment decisions. As this advice is typically imperfect, decision-makers have to exert discretion as to whether actually follow that advice: they have to “appropriately” rely on correct and turn down incorrect advice. However, current research on appropriate reliance still lacks a common definition as well as an operational measurement concept. Additionally, no in-depth behavioral experiments have been conducted that help understand the factors influencing this behavior. In this paper, we propose Appropriateness of Reliance (AoR) as an underlying, quantifiable two-dimensional measurement concept. We develop a research model that analyzes the effect of providing explanations for AI advice. In an experiment with 200 participants, we demonstrate how these explanations influence the AoR, and, thus, the effectiveness of AI advice. Our work contributes fundamental concepts for the analysis of reliance behavior and the purposeful design of AI advisors.

Overview

CoChat Summary

TL;DR

The paper proposes Appropriateness of Reliance (AoR) as a two-dimensional metric—relative AI reliance (RAIR) and relative self-reliance (RSR)—for measuring whether decision-makers appropriately follow correct AI advice and reject incorrect AI advice. In a 200-participant hotel-review experiment, feature-importance explanations significantly increased RAIR but not RSR, and the human-AI team did not reach complementary team performance, so appropriate reliance was not displayed.

Key findings

  • AoR is defined as the tuple of RAIR and RSR; appropriate reliance (AR) is defined as any AoR tuple that leads to complementary team performance (CTP).
  • In the control condition, RAIR was 29.59% and RSR was 71.87%; with feature-importance explanations, RAIR rose significantly to 38.87% (t = -1.95, p = 0.05), while RSR did not change significantly (69.45%; t = 0.61, p = 0.54).
  • Explanations significantly increased the change in self-confidence (t = -2.33, p = 0.02), which partially mediated the effect on RAIR; explanations had no significant effect on trust (H4 not supported), but trust increased RAIR and decreased RSR (H5a and H5b supported).
  • CTP was not reached: down-sampled AI performance was 50% in both conditions; human accuracy was 55.50% (control) and 53.85% (feature importance), and AI-assisted accuracy was 53.94% (control) and 56.30% (feature importance), with human-AI team performance not significantly different from human accuracy.

Methods

  • Between-subject online experiment with 200 participants recruited via Prolific; one participant in the feature-importance condition was excluded for a failed manipulation check, and SEM analyses used a final sample of 199.
  • Participants classified 16 deceptive or genuine hotel reviews sequentially: first without AI advice and with a confidence rating, then after receiving AI advice (SVM with 86% accuracy) either alone or with LIME feature-importance explanations, then revised their decision and confidence.
  • Analysis used descriptive t-tests and structural equation modeling (SEM) with the MLR estimator in lavaan, measuring RAIR, RSR, trust, and change in self-confidence.

Limitations

  • The paper states that deception detection is difficult for humans, that generalizability is limited by the choice of explanations, that the concept is limited to classification tasks, and that the sequential task setup may change the task, induce anchoring, reduce over-reliance, and may not always be possible or desired in real-world settings; crowd workers also limit generalizability.
  • Evident from the text: the sample was relatively young (mean age 27.5, SD 8.5) and drawn from an online platform, and the experimental AI’s down-sampled performance was 50% in both conditions, which may constrain conclusions about CTP and appropriate reliance.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3581641.3584066

Cites

83 of 83 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Combinatorics of initial human decisions, AI advice and human reliance for a single task instance in a sequential task setting.
Fig. 1 Combinatorics of initial human decisions, AI advice and human reliance for a single task instance in a sequential task setting.From the PDF
Fig. 2 Two-dimensional depiction of the Appropriateness of Reliance (AoR).
Fig. 2 Two-dimensional depiction of the Appropriateness of Reliance (AoR).From the PDF
Fig. 3 Illustrative example of how humans can increase their relative AI reliance ( 𝑅𝐴𝐼𝑅 ) based on explanations of the AI advice.
Fig. 3 Illustrative example of how humans can increase their relative AI reliance ( 𝑅𝐴𝐼𝑅 ) based on explanations of the AI advice.From the PDF
Fig. 4 Research model on the effect of explanations on Appropriateness of Reliance (AoR).
Fig. 4 Research model on the effect of explanations on Appropriateness of Reliance (AoR).From the PDF
Fig. 5 Online experiment graphical user interface for the feature importance condition. The ground truth of the exemplarily shown hotel review is 'fake'. The design of the interface is adapted from Lai et al. [42].
Fig. 5 Online experiment graphical user interface for the feature importance condition. The ground truth of the exemplarily shown hotel review is 'fake'. The design of the interface is adapted from Lai et al. [42].From the PDF
Fig. 6 Illustration of Appropriateness of Reliance (AoR) including standard errors. Explanations increase the 𝑅𝐴𝐼𝑅 significantly. Differences in 𝑅𝑆𝑅 are not significant.
Fig. 6 Illustration of Appropriateness of Reliance (AoR) including standard errors. Explanations increase the 𝑅𝐴𝐼𝑅 significantly. Differences in 𝑅𝑆𝑅 are not significant.From the PDF
Fig. 7 Structural equation modeling results. Significance: ★★★ p < 0.01, ★★ p <0.05, ★ p < 0.1.
Fig. 7 Structural equation modeling results. Significance: ★★★ p < 0.01, ★★ p <0.05, ★ p < 0.1.From the PDF

Verification

Claims the paper makes

17 Valid · 3 Not valid · 10 Not checked · 54 Own results and interpretations

29 more claims · 5 with their cited sources checked

Create a free account to see all 84 claims and to verify the 2 references still open.

See all claims — free account
  • It may state two separate claims as one. 21
  • Probably not a substantial claim. 33
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 43 references

Research works printed but not linked: 3

  1. [29] Patrick Hemmer, Max Schemmer, Michael Vössing, and Niklas Kühl. 2021. Human-AI Complementarity in Hybrid Intelligence Systems: A Structured Literature Review. PACIS 2021 Proceedings (2021).
  2. [66] Keng Siau and Weiyu Wang. 2018. Building trust in artificial intelligence, machine learning, and robotics. Cutter business technology journal 31, 2 (2018), 47–53.
  3. [68] Philipp Spitzer, Niklas Kühl, and Marc Goutier. 2022. Training Novices: The Role of Human-AI Collaboration and Knowledge Transfer. Workshop on Human-Machine Collaboration and Teaming (HM-CaT 2022). The 39th International Conference on Machine Learning, Baltimore, Maryland, USA (2022).

Cited by

See all 208

Metadata

Authors
Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G.
Affiliations
Karlsruhe Institute of Technology (DE)
Venue
Proceedings of the 28th International Conference on Intelligent User Interfaces
Publisher
ACM
Published
2023-03-27
Pages
410-422
OpenAlex
W4360991389
Cited by
208
References
43
Citations by year
2026: 93 · 2025: 71 · 2024: 37 · 2023: 6
FWCI
36.59
Type
conference paper
Language
en
Open access
Locations
  • ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • Repository KITopen (Karlsruhe Institute of Technology) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
  • KITopen
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
Keywords
AI advice, AI explanations, human-AI decision-making, trust in AI, measurement concepts, investment decision-making, treatment decision-making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations. https://app.cochat.ai/papers/10.1145/3581641.3584066
The paper
Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G. (2023). Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations. Proceedings of the 28th International Conference on Intelligent User Interfaces. https://doi.org/10.1145/3581641.3584066

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.