The DOI matches its Crossref record; 83 of the 83 references it cites were found in a registry or an index; 23 of the 39 sources its claims cite were checked for support.
Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
Open accessAbstract
AI advice is becoming increasingly popular, e.g., in investment and medical treatment decisions. As this advice is typically imperfect, decision-makers have to exert discretion as to whether actually follow that advice: they have to “appropriately” rely on correct and turn down incorrect advice. However, current research on appropriate reliance still lacks a common definition as well as an operational measurement concept. Additionally, no in-depth behavioral experiments have been conducted that help understand the factors influencing this behavior. In this paper, we propose Appropriateness of Reliance (AoR) as an underlying, quantifiable two-dimensional measurement concept. We develop a research model that analyzes the effect of providing explanations for AI advice. In an experiment with 200 participants, we demonstrate how these explanations influence the AoR, and, thus, the effectiveness of AI advice. Our work contributes fundamental concepts for the analysis of reliance behavior and the purposeful design of AI advisors.
Overview
CoChat Summary
TL;DR
The paper proposes Appropriateness of Reliance (AoR) as a two-dimensional metric—relative AI reliance (RAIR) and relative self-reliance (RSR)—for measuring whether decision-makers appropriately follow correct AI advice and reject incorrect AI advice. In a 200-participant hotel-review experiment, feature-importance explanations significantly increased RAIR but not RSR, and the human-AI team did not reach complementary team performance, so appropriate reliance was not displayed.
Key findings
- AoR is defined as the tuple of RAIR and RSR; appropriate reliance (AR) is defined as any AoR tuple that leads to complementary team performance (CTP).
- In the control condition, RAIR was 29.59% and RSR was 71.87%; with feature-importance explanations, RAIR rose significantly to 38.87% (t = -1.95, p = 0.05), while RSR did not change significantly (69.45%; t = 0.61, p = 0.54).
- Explanations significantly increased the change in self-confidence (t = -2.33, p = 0.02), which partially mediated the effect on RAIR; explanations had no significant effect on trust (H4 not supported), but trust increased RAIR and decreased RSR (H5a and H5b supported).
- CTP was not reached: down-sampled AI performance was 50% in both conditions; human accuracy was 55.50% (control) and 53.85% (feature importance), and AI-assisted accuracy was 53.94% (control) and 56.30% (feature importance), with human-AI team performance not significantly different from human accuracy.
Methods
- Between-subject online experiment with 200 participants recruited via Prolific; one participant in the feature-importance condition was excluded for a failed manipulation check, and SEM analyses used a final sample of 199.
- Participants classified 16 deceptive or genuine hotel reviews sequentially: first without AI advice and with a confidence rating, then after receiving AI advice (SVM with 86% accuracy) either alone or with LIME feature-importance explanations, then revised their decision and confidence.
- Analysis used descriptive t-tests and structural equation modeling (SEM) with the MLR estimator in lavaan, measuring RAIR, RSR, trust, and change in self-confidence.
Limitations
- The paper states that deception detection is difficult for humans, that generalizability is limited by the choice of explanations, that the concept is limited to classification tasks, and that the sequential task setup may change the task, induce anchoring, reduce over-reliance, and may not always be possible or desired in real-world settings; crowd workers also limit generalizability.
- Evident from the text: the sample was relatively young (mean age 27.5, SD 8.5) and drawn from an online platform, and the experimental AI’s down-sampled performance was 50% in both conditions, which may constrain conclusions about CTP and appropriate reliance.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




![Fig. 5 Online experiment graphical user interface for the feature importance condition. The ground truth of the exemplarily shown hotel review is 'fake'. The design of the interface is adapted from Lai et al. [42].](https://figures.cochat.ai/paper-figures/aa69c3bc1d844572931891d4e00e8f4d/appropriate-reliance-on-ai-advice-conceptualization-and-the-effect-of-figure-5.png)


Verification
Claims the paper makes
17 Valid · 3 Not valid · 10 Not checked · 54 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The sequential task setup could induce an anchoring effect which prevents the human to more actively take the AI into account [9].”
What the sources say
- Disputedcites [9] [9] Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1-21. - sourceExperimental — accuracy not yet measured
Section 3.2.2 (p. 7) describes a sequence in which people decide before seeing the AI and hypothesizes that it will motivate them to engage with the AI explanation, not prevent engagement. Section 2.2 (p. 4) instead associates anchoring with showing the AI recommendation before the person makes an independent decision. (LLM, conf=0.84)
Checked by a reasoning model — may be wrong.
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“Understanding is crucial in building trust [23].”
What the sources say
- Supportedcites [23] [23] Leilani H Gilpin, Cecilia Testart, Nathaniel Fruchter, and Julius Adebayo. 2019. Explaining explanations to society. arXiv preprint arXiv:1901.06560 (2019). - source
The Discussion and Conclusion (p. 5) says that truly trusting AI systems requires people to feel confident they understand how decisions are made. It also says they need recourse, so understanding is crucial but not sufficient on its own. (LLM, conf=0.92)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Confidence in one's own decision is a key mechanism underlying advice acceptance [14, 73].”
What the sources say
- Supportedcites [14] [14] Leah Chong, Guanglu Zhang, Kosa Goucher-Lambert, Kenneth Kotovsky, and Jonathan Cagan. 2022. Human confidence in artificial intelligence and in themselves: The evolution and impact of confidence on adoption of AI advice. Computers in Human Behavior 127 (2022), 107018. - source
The Discussion reports that self-confidence significantly contributes to whether people accept AI suggestions and describes it as a driving factor. The Conclusion likewise reports a significant impact of self-confidence on accepting or rejecting those suggestions. (LLM, conf=0.90)
- Supportedcites [73] [73] Xiuxin Wang and Xiufang Du. 2018. Why does advice discounting occur? The combined roles of confidence and trust. Frontiers in Psychology 9 (2018), 2381. - source
The Introduction calls confidence in one’s own judgment a key mechanism underlying advice taking. The General Discussion reports that advice discounting decreased when confidence decreased and that confidence helped predict advice taking. Page numbers are not provided in our copy. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We would like to emphasize that deception detection is a difficult task for humans [42, 43].”
What the sources say
- Supportedcites [42] [42] Vivian Lai, Han Liu, and Chenhao Tan. 2020. " Why is' Chicago'deceptive?" Towards Building Model-Driven Tutorials for Humans. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems . 1-13. - source
The Deception detection section (p. 3) says it is well recognized that humans struggle to detect deception. The Discussion (p. 10) likewise describes deception detection as potentially too challenging for humans. (LLM, conf=0.94)
- Supportedcites [43] [43] Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency . 29-38. - source
The paper explicitly calls deception detection “a challenging task for humans” in both its introductory discussion (p. 2) and Related Work (p. 3). This directly supports the citing sentence’s paraphrase. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“While this may sound disappointing at first, it also shows that the claim that explanations would reduce overreliance [6, 9] does not seem to hold for all kinds of tasks.”
What the sources say
- Supportedcites [6] [6] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1-16. - source
The Introduction (p. 2) reports that explanations increased reliance on AI recommendations even when the AI was incorrect. Section 6.2 (p. 12) describes this as increased blind trust rather than appropriate reliance, supporting the qualified claim that explanations do not reduce overreliance in all tasks. (LLM, conf=0.93)
- Supportedcites [9] [9] Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1-21. - source
The Introduction (p. 2) says explanations were intended to reduce overreliance but have not had substantial success. The Discussion (p. 16) reports that, in this paper’s decision-making task, participants using simple explainable AI still followed incorrect suggestions. This supports the qualified claim that explanations do not reduce overreliance in all tasks. (LLM, conf=0.93)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
29 more claims · 5 with their cited sources checked
Create a free account to see all 84 claims and to verify the 2 references still open.
- It may state two separate claims as one. 21
- Probably not a substantial claim. 33
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Research on human-AI decision making suggests that reliance is appropriate when people accept correct AI output and reject incorrect output; thus, the relevant outcome is not simply whether and how often students defer to AI, but how well their reliance tracks the quality of AI suggestions (Raees et al., 2026;Schemmer et al., 2023;Bansal et al., 2021).”
From Think Before You Accept: Can Written Justification Reduce Uncritical Uptake of AI Writing Suggestions? (2026) · cites this paper as Schemmer et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Research on human-AI decision making suggests that reliance is appropriate when people accept correct AI output and reject incorrect output; thus, the relevant outcome is not simply whether and how often students defer to AI, but how well their reliance tracks the quality of AI suggestions (Raees et al., 2026;Schemmer et al., 2023;Bansal et al., 2021).”
From Think Before You Accept: Can Written Justification Reduce Uncritical Uptake of AI Writing Suggestions? (2026) · cites this paper as Schemmer et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“in contrast to prior work which has evaluated reliance in action space only [3, 28, 35].”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [28]
Citations
Cites
Hu, L., & Bentler, P. M. (1999) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705519909540118 · Cited by 108,788
lavaan : An R Package for Structural Equation Modeling
Rosseel, Y. (2012) · Journal of Statistical Software · doi:10.18637/jss.v048.i02 · Cited by 27,955
Comparative fit indexes in structural models.
Bentler, P. M. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.107.2.238 · Cited by 25,266
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
The future of employment: How susceptible are jobs to computerisation?
Frey, C. B., & Osborne, M. A. (2017) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2016.08.019 · Cited by 8,727
Trust and TAM in Online Shopping: An Integrated Model1
Gefen, D., Karahanna, E., & Straub, D. W. (2003) · MIS Quarterly · doi:10.2307/30036519 · Cited by 8,635
An Examination of the Nature of Trust in Buyer-Seller Relationships
Doney, P. M., & Cannon, J. P. (1997) · Journal of Marketing · doi:10.2307/1251829 · Cited by 6,822 · cited 1×
“In our study, we define trust as a belief in the integrity, benevolence, trustworthiness, and predictability of the AI advisor following [15, 17, 21, 49].”
BackgroundDeterminants of Long-Term Orientation in Buyer-Seller Relationships
Ganesan, S. (1994) · Journal of Marketing · doi:10.1177/002224299405800201 · Cited by 6,741 · cited 1×
“In our study, we define trust as a belief in the integrity, benevolence, trustworthiness, and predictability of the AI advisor following [15, 17, 21, 49].”
BackgroundTrust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 3×
“Most researchers that studied AR so far have proposed to use explanations of AI as a means for AR [1, 43, 78].”
Background
Research works printed but not linked: 3
- [29] Patrick Hemmer, Max Schemmer, Michael Vössing, and Niklas Kühl. 2021. Human-AI Complementarity in Hybrid Intelligence Systems: A Structured Literature Review. PACIS 2021 Proceedings (2021).
- [66] Keng Siau and Weiyu Wang. 2018. Building trust in artificial intelligence, machine learning, and robotics. Cutter business technology journal 31, 2 (2018), 47–53.
- [68] Philipp Spitzer, Niklas Kühl, and Marc Goutier. 2022. Training Novices: The Role of Human-AI Collaboration and Knowledge Transfer. Workshop on Human-Machine Collaboration and Teaming (HM-CaT 2022). The 39th International Conference on Machine Learning, Baltimore, Maryland, USA (2022).
Cited by
Gomez, C., Cho, S. M., Ke, S., Huang, C.-M., & Unberath, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1521066 · Cited by 209
Enhancing AI-Assisted Group Decision Making through LLM-Powered Devil's Advocate
Chiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645199 · Cited by 142
“For example, many studies specifically examine whether people exhibit any over-reliance or under-reliance on AI recommendations [11, 15, 23, 33, 37, 45, 53, 56, 72, 76, 84, 98, 99, 105, 109, 110].”
BackgroundMehrotra, S., Degachi, C., Vereschak, O., Jonker, C. M., & Tielman, M. L. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3696449 · Cited by 254
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147
Explainable AI improves task performance in human–AI collaboration
Senoner, J., Schallmoser, S., Kratzwald, B., Feuerriegel, S., & Netland, T. H. (2024) · Scientific Reports · doi:10.1038/s41598-024-82501-9 · Cited by 199
“Unlike previous works on studying task performance in human–AI collaboration 41–44 , we (i) conducted experiments of two real-world job tasks in manufacturing and medicine and (ii) recruited domain experts for those tasks, i.e., factory workers and radiologists.”
Metadata
- Authors
- Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G.
- Affiliations
- Karlsruhe Institute of Technology (DE)
- Venue
- Proceedings of the 28th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2023-03-27
- Pages
- 410-422
- OpenAlex
- W4360991389
- Semantic Scholar
- 95f751e8535a6d7f8fe733ab4729f738ef0566c4
- Cited by
- 208
- References
- 43
- Citations by year
- 2026: 93 · 2025: 71 · 2024: 37 · 2023: 6
- FWCI
- 36.59
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- Repository KITopen (Karlsruhe Institute of Technology) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- KITopen
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- AI advice, AI explanations, human-AI decision-making, trust in AI, measurement concepts, investment decision-making, treatment decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations. https://app.cochat.ai/papers/10.1145/3581641.3584066
- The paper
- Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G. (2023). Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations. Proceedings of the 28th International Conference on Intelligent User Interfaces. https://doi.org/10.1145/3581641.3584066
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.