Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
Open accessThe DOI matches its Crossref record; 83 of the 83 references it cites were found in a registry or an index; 23 of the 39 sources its claims cite were checked for support.
Abstract
AI advice is becoming increasingly popular, e.g., in investment and medical treatment decisions. As this advice is typically imperfect, decision-makers have to exert discretion as to whether actually follow that advice: they have to “appropriately” rely on correct and turn down incorrect advice. However, current research on appropriate reliance still lacks a common definition as well as an operational measurement concept. Additionally, no in-depth behavioral experiments have been conducted that help understand the factors influencing this behavior. In this paper, we propose Appropriateness of Reliance (AoR) as an underlying, quantifiable two-dimensional measurement concept. We develop a research model that analyzes the effect of providing explanations for AI advice. In an experiment with 200 participants, we demonstrate how these explanations influence the AoR, and, thus, the effectiveness of AI advice. Our work contributes fundamental concepts for the analysis of reliance behavior and the purposeful design of AI advisors.
Overview
CoChat Summary
TL;DR
The paper proposes Appropriateness of Reliance (AoR) as a two-dimensional metric—relative AI reliance (RAIR) and relative self-reliance (RSR)—for measuring whether decision-makers appropriately follow correct AI advice and reject incorrect AI advice. In a 200-participant hotel-review experiment, feature-importance explanations significantly increased RAIR but not RSR, and the human-AI team did not reach complementary team performance, so appropriate reliance was not displayed.
Key findings
- AoR is defined as the tuple of RAIR and RSR; appropriate reliance (AR) is defined as any AoR tuple that leads to complementary team performance (CTP).
- In the control condition, RAIR was 29.59% and RSR was 71.87%; with feature-importance explanations, RAIR rose significantly to 38.87% (t = -1.95, p = 0.05), while RSR did not change significantly (69.45%; t = 0.61, p = 0.54).
- Explanations significantly increased the change in self-confidence (t = -2.33, p = 0.02), which partially mediated the effect on RAIR; explanations had no significant effect on trust (H4 not supported), but trust increased RAIR and decreased RSR (H5a and H5b supported).
- CTP was not reached: down-sampled AI performance was 50% in both conditions; human accuracy was 55.50% (control) and 53.85% (feature importance), and AI-assisted accuracy was 53.94% (control) and 56.30% (feature importance), with human-AI team performance not significantly different from human accuracy.
Methods
- Between-subject online experiment with 200 participants recruited via Prolific; one participant in the feature-importance condition was excluded for a failed manipulation check, and SEM analyses used a final sample of 199.
- Participants classified 16 deceptive or genuine hotel reviews sequentially: first without AI advice and with a confidence rating, then after receiving AI advice (SVM with 86% accuracy) either alone or with LIME feature-importance explanations, then revised their decision and confidence.
- Analysis used descriptive t-tests and structural equation modeling (SEM) with the MLR estimator in lavaan, measuring RAIR, RSR, trust, and change in self-confidence.
Limitations
- The paper states that deception detection is difficult for humans, that generalizability is limited by the choice of explanations, that the concept is limited to classification tasks, and that the sequential task setup may change the task, induce anchoring, reduce over-reliance, and may not always be possible or desired in real-world settings; crowd workers also limit generalizability.
- Evident from the text: the sample was relatively young (mean age 27.5, SD 8.5) and drawn from an online platform, and the experimental AI’s down-sampled performance was 50% in both conditions, which may constrain conclusions about CTP and appropriate reliance.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




![Fig. 5 Online experiment graphical user interface for the feature importance condition. The ground truth of the exemplarily shown hotel review is 'fake'. The design of the interface is adapted from Lai et al. [42].](https://figures.cochat.ai/paper-figures/aa69c3bc1d844572931891d4e00e8f4d/appropriate-reliance-on-ai-advice-conceptualization-and-the-effect-of-figure-5.png)


Verification
Claims the paper makes
17 Valid · 3 Not valid · 10 Not checked · 54 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“The sequential task setup could induce an anchoring effect which prevents the human to more actively take the AI into account [9].”
What the sources say
- Disputedcites [9] [9] Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1-21. - sourceExperimental — accuracy not yet measured
Section 3.2.2 (p. 7) describes a sequence in which people decide before seeing the AI and hypothesizes that it will motivate them to engage with the AI explanation, not prevent engagement. Section 2.2 (p. 4) instead associates anchoring with showing the AI recommendation before the person makes an independent decision. (LLM, conf=0.84)
Checked by a reasoning model — may be wrong.
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“Understanding is crucial in building trust [23].”
What the sources say
- Supportedcites [23] [23] Leilani H Gilpin, Cecilia Testart, Nathaniel Fruchter, and Julius Adebayo. 2019. Explaining explanations to society. arXiv preprint arXiv:1901.06560 (2019). - source
The Discussion and Conclusion (p. 5) says that truly trusting AI systems requires people to feel confident they understand how decisions are made. It also says they need recourse, so understanding is crucial but not sufficient on its own. (LLM, conf=0.92)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Confidence in one's own decision is a key mechanism underlying advice acceptance [14, 73].”
What the sources say
- Supportedcites [14] [14] Leah Chong, Guanglu Zhang, Kosa Goucher-Lambert, Kenneth Kotovsky, and Jonathan Cagan. 2022. Human confidence in artificial intelligence and in themselves: The evolution and impact of confidence on adoption of AI advice. Computers in Human Behavior 127 (2022), 107018. - source
The Discussion reports that self-confidence significantly contributes to whether people accept AI suggestions and describes it as a driving factor. The Conclusion likewise reports a significant impact of self-confidence on accepting or rejecting those suggestions. (LLM, conf=0.90)
- Supportedcites [73] [73] Xiuxin Wang and Xiufang Du. 2018. Why does advice discounting occur? The combined roles of confidence and trust. Frontiers in Psychology 9 (2018), 2381. - source
The Introduction calls confidence in one’s own judgment a key mechanism underlying advice taking. The General Discussion reports that advice discounting decreased when confidence decreased and that confidence helped predict advice taking. Page numbers are not provided in our copy. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We would like to emphasize that deception detection is a difficult task for humans [42, 43].”
What the sources say
- Supportedcites [42] [42] Vivian Lai, Han Liu, and Chenhao Tan. 2020. " Why is' Chicago'deceptive?" Towards Building Model-Driven Tutorials for Humans. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems . 1-13. - source
The Deception detection section (p. 3) says it is well recognized that humans struggle to detect deception. The Discussion (p. 10) likewise describes deception detection as potentially too challenging for humans. (LLM, conf=0.94)
- Supportedcites [43] [43] Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency . 29-38. - source
The paper explicitly calls deception detection “a challenging task for humans” in both its introductory discussion (p. 2) and Related Work (p. 3). This directly supports the citing sentence’s paraphrase. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“While this may sound disappointing at first, it also shows that the claim that explanations would reduce overreliance [6, 9] does not seem to hold for all kinds of tasks.”
What the sources say
- Supportedcites [6] [6] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1-16. - source
The Introduction (p. 2) reports that explanations increased reliance on AI recommendations even when the AI was incorrect. Section 6.2 (p. 12) describes this as increased blind trust rather than appropriate reliance, supporting the qualified claim that explanations do not reduce overreliance in all tasks. (LLM, conf=0.93)
- Supportedcites [9] [9] Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1-21. - source
The Introduction (p. 2) says explanations were intended to reduce overreliance but have not had substantial success. The Discussion (p. 16) reports that, in this paper’s decision-making task, participants using simple explainable AI still followed incorrect suggestions. This supports the qualified claim that explanations do not reduce overreliance in all tasks. (LLM, conf=0.93)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
29 more claims · 5 with their cited sources checked
Create a free account to see all 84 claims and to verify the 2 references still open.
- It may state two separate claims as one. 21
- Probably not a substantial claim. 33
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Research on human-AI decision making suggests that reliance is appropriate when people accept correct AI output and reject incorrect output; thus, the relevant outcome is not simply whether and how often students defer to AI, but how well their reliance tracks the quality of AI suggestions (Raees et al., 2026;Schemmer et al., 2023;Bansal et al., 2021).”
From Think Before You Accept: Can Written Justification Reduce Uncritical Uptake of AI Writing Suggestions? (2026) · cites this paper as Schemmer et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“Research on human-AI decision making suggests that reliance is appropriate when people accept correct AI output and reject incorrect output; thus, the relevant outcome is not simply whether and how often students defer to AI, but how well their reliance tracks the quality of AI suggestions (Raees et al., 2026;Schemmer et al., 2023;Bansal et al., 2021).”
From Think Before You Accept: Can Written Justification Reduce Uncritical Uptake of AI Writing Suggestions? (2026) · cites this paper as Schemmer et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“in contrast to prior work which has evaluated reliance in action space only [3, 28, 35].”
From A Decision Theoretic Framework for Measuring AI Reliance (2024) · cites this paper as [28]
Citations
Cites
Hu, L., & Bentler, P. M. (1999) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705519909540118 · Cited by 108,788
lavaan : An R Package for Structural Equation Modeling
Rosseel, Y. (2012) · Journal of Statistical Software · doi:10.18637/jss.v048.i02 · Cited by 27,955
Comparative fit indexes in structural models.
Bentler, P. M. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.107.2.238 · Cited by 25,266
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
The future of employment: How susceptible are jobs to computerisation?
Frey, C. B., & Osborne, M. A. (2017) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2016.08.019 · Cited by 8,727
Trust and TAM in Online Shopping: An Integrated Model1
Gefen, D., Karahanna, E., & Straub, D. W. (2003) · MIS Quarterly · doi:10.2307/30036519 · Cited by 8,635
An Examination of the Nature of Trust in Buyer-Seller Relationships
Doney, P. M., & Cannon, J. P. (1997) · Journal of Marketing · doi:10.2307/1251829 · Cited by 6,822 · cited 1×
“In our study, we define trust as a belief in the integrity, benevolence, trustworthiness, and predictability of the AI advisor following [15, 17, 21, 49].”
BackgroundDeterminants of Long-Term Orientation in Buyer-Seller Relationships
Ganesan, S. (1994) · Journal of Marketing · doi:10.1177/002224299405800201 · Cited by 6,741 · cited 1×
“In our study, we define trust as a belief in the integrity, benevolence, trustworthiness, and predictability of the AI advisor following [15, 17, 21, 49].”
BackgroundTrust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 3×
“Most researchers that studied AR so far have proposed to use explanations of AI as a means for AR [1, 43, 78].”
BackgroundRelationship Quality in Services Selling: An Interpersonal Influence Perspective
Crosby, L. A., Evans, K. R., & Cowles, D. (1990) · Journal of Marketing · doi:10.1177/002224299005400306 · Cited by 4,763 · cited 1×
“In our study, we define trust as a belief in the integrity, benevolence, trustworthiness, and predictability of the AI advisor following [15, 17, 21, 49].”
BackgroundInternational evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287 · cited 1×
“For example, AI advises medical professionals in breast cancer screening [48], or in loan decisions[16].”
BackgroundHarrison McKnight, D., Choudhury, V., & Kacmar, C. (2002) · The Journal of Strategic Information Systems · doi:10.1016/s0963-8687(02)00020-3 · Cited by 2,164
Shin, D. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102551 · Cited by 1,548 · cited 2×
“Research has already investigated in-depth the effect of explanations on AI advice acceptance [65].”
BackgroundTrust, control strategies and allocation of function in human-machine systems
LEE, J., & MORAY, N. (1992) · Ergonomics · doi:10.1080/00140139208967392 · Cited by 1,459 · cited 2×
“In 1992, Lee and Moray [44], already discussed the influence of self-confidence and”
BackgroundConsistent Individualized Feature Attribution for Tree Ensembles
Lundberg, S. M., Erion, G. G., & Lee, S.-I. (2018) · arXiv · doi:10.48550/arxiv.1802.03888 · Cited by 1,430
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276 · cited 1×
“Reliance itself is defined as a behavior [18, 45].”
BackgroundAutomation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162 · cited 1×
“on automation [4, 24, 54] and, therefore, should decrease the RSR.”
BackgroundBuçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Bonaccio, S., & Dalal, R. S. (2006) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2006.07.001 · Cited by 1,011 · cited 2×
“The term “judge” refers to the decision-maker who receives the advice and must decide what to do with it [8].”
BackgroundFinding Deceptive Opinion Spam by Any Stretch of the Imagination
Ott, M., Choi, Y., Cardie, C., & Hancock, J. T. (2011) · arXiv · doi:10.48550/arxiv.1107.4557 · Cited by 1,008
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895 · cited 3×
“Recently, a new line of research has emerged that argues that maximising AI reliance does not fully exploit the potential of state-of-the-art human-AI decision making [6, 9, 62, 78].”
BackgroundPerformance Consequences of Automation-Induced 'Complacency'
Parasuraman, R., Molloy, R., & Singh, I. L. (1993) · The International Journal of Aviation Psychology · doi:10.1207/s15327108ijap0301_1 · Cited by 875 · cited 1×
“on automation [4, 24, 54] and, therefore, should decrease the RSR.”
BackgroundProceedings of the 18th European conference on information systems (ECIS2010)
Tervonen, T., Brock, D. B., Graeff, D. P., & Hillege, H. L. (2010) · Cited by 836
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 1×
“recommendation, especially in cases where the AI advice is wrong [6, 57, 63].”
BackgroundManipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Model-Agnostic Interpretability of Machine Learning
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1606.05386 · Cited by 699 · cited 1×
“For the explanations, we use a state-of-the-art explanation technique, LIME feature importance explanations [58], as it is the most common one for textual data.”
MethodVanderwende, L., Daumé, H., & Kirchhoff, K. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 685
Receiving other people’s advice: Influence and benefit
Yaniv, I. (2004) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2003.08.002 · Cited by 652
Taking Advice: Accepting Help, Improving Judgment, and Sharing Responsibility
Harvey, N., & Fischer, I. (1997) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1997.2697 · Cited by 588
Explanation, imagination, and confidence in judgment.
Koehler, D. J. (1991) · Psychological Bulletin · doi:10.1037//0033-2909.110.3.499 · Cited by 561
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445923 · Cited by 545
Trust, Confidence, and Expertise in a Judge-Advisor System
Sniezek, J. A., & Van Swol, L. M. (2001) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.2000.2926 · Cited by 469
Will Humans-in-the-Loop Become Borgs? Merits and Pitfalls of Working with AI
Fügener, A., Grahl, J., Gupta, A., & Ketter, W. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16553 · Cited by 469
The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441 · cited 1×
“Bussone et al. [10] assess how explanations impact trust and reliance on clinical decision support systems.”
BackgroundJussupow, E., Spohrer, K., Heinzl, A., & Gawlitza, J. (2021) · Information Systems Research · doi:10.1287/isre.2020.0980 · Cited by 422 · cited 3×
“the reliance [35], trust [66], utilization [2], compliance [37], or acceptance [65] of AI advice [34].”
BackgroundEvaluating XAI: A comparison of rule-based and example-based explanations
van der Waa, J., Nieuwburg, E., Cremers, A., & Neerincx, M. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2020.103404 · Cited by 393
Lai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377
Wang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320 · cited 9×
“In this work, we focus on AR with respect to performance ( 𝑃 ) following Talone [69], Wang and Yin [74].”
BackgroundMethodResultInfluentialConfidence, uncertainty, and the use of information.
Peterson, D. K., & Pitz, G. F. (1988) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037//0278-7393.14.1.85 · Cited by 278 · cited 1×
“Confidence is defined as a person’s degree of belief that their own decision is correct [55, 77].”
BackgroundChong, L., Zhang, G., Goucher-Lambert, K., Kotovsky, K., & Cagan, J. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2021.107018 · Cited by 262 · cited 1×
“Confidence in one’s own decision is a key mechanism underlying advice acceptance [14, 73].”
BackgroundRobo-Advisory: Digitalization and Automation of Financial Advisory
Jung, D., Dorner, V., Glaser, F., & Morana, S. (2018) · Business & Information Systems Engineering · doi:10.1007/s12599-018-0521-9 · Cited by 254
Bailey, N. R., & Scerbo, M. W. (2007) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220500535301 · Cited by 224 · cited 1×
“on automation [4, 24, 54] and, therefore, should decrease the RSR.”
BackgroundHow do visual explanations foster end users' appropriate trust in machine learning?
Yang, F., Huang, Z., Scholtz, J., & Arendt, D. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377480 · Cited by 200 · cited 3×
“[6] Accuracy on correct or incorrect AI advice Local feature importance Negative Adaptive explanations Positive based on AI confidence [9] Ratio of reliance on incorrect AI advice Local feature importance Negative [32] Ratio of reliance on correct Local feature importance No effect or incorrect AI advice Predictive ou…”
BackgroundResultLai, K. (2018) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705511.2017.1392248 · Cited by 190 · cited 1×
“Therefore, we use the MLR estimator [39].”
The Social Influence of Confidence in Group Decision Making
Zarnoth, P., & Sniezek, J. A. (1997) · Journal of Experimental Social Psychology · doi:10.1006/jesp.1997.1326 · Cited by 168 · cited 1×
“Confidence is defined as a person’s degree of belief that their own decision is correct [55, 77].”
BackgroundLet Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems
Kunkel, J., Donkers, T., Michael, L., Barbu, C.-M., & Ziegler, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300717 · Cited by 156
Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Lai, V., Carton, S., Bhatnagar, R., Liao, Q. V., Zhang, Y., & Tan, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501999 · Cited by 145 · cited 1×
“Third, as modern AI is not only more performant and offers more application areas, but also complements humans [27, 28, 40], there is an increasing potential to achieve complementary team performance (CTP), i.”
Background"Why is 'Chicago' deceptive?" Towards Building Model-Driven Tutorials for Humans
Lai, V., Liu, H., & Tan, C. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376873 · Cited by 123
(2018) · doi:10.1201/9781315137957-2 · Cited by 108
Does Explainable Artificial Intelligence Improve Human Decision-Making?
Alufaisan, Y., Marusich, L. R., Bakdash, J. Z., Zhou, Y., & Kantarcioglu, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16819 · Cited by 106
Before and beyond trust: reliance in medical AI
Kerasidou, C. (., Kerasidou, A., Buscher, M., & Wilkinson, S. (2022) · Journal of Medical Ethics · doi:10.1136/medethics-2020-107095 · Cited by 95
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AIPreprint
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.07487 · Cited by 81
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies
Lai, V., Chen, C., Liao, Q. V., Smith-Renner, A., & Tan, C. (2021) · arXiv · doi:10.48550/arxiv.2112.11471 · Cited by 75 · cited 5×
“1It is worth to mention that several studies have examined AR in automation and robotics research [45, 72], but an agreed-upon definition of AR and a respective metric are still missing [41, 72].”
BackgroundInfluentialDo ‘big losses’ in judgmental adjustments to statistical forecasts affect experts’ behaviour?
Petropoulos, F., Fildes, R., & Goodwin, P. (2016) · European Journal of Operational Research · doi:10.1016/j.ejor.2015.06.002 · Cited by 71
Hemmer, P., Schellhammer, S., Vossing, M., et al. (2022) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.2206.07948 · Cited by 64
Feature-Based Explanations Don't Help People Detect Misclassifications of Online Toxicity
Carton, S., Mei, Q., & Resnick, P. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7282 · Cited by 58 · cited 1×
“[3, 9, 11, 71].”
Do explanations make VQA models more predictable to a human?
Chandrasekaran, A., Prabhu, V., Yadav, D., Chattopadhyay, P., & Parikh, D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1128 · Cited by 55 · cited 1×
“[12] analyze whether humans can learn to predict the AI’s behavior.”
BackgroundYou’d Better Stop! Understanding Human Reliance on Machine Learning Models under Covariate Shift
Chiang, C.-W., & Yin, M. (2021) · 13th ACM Web Science Conference 2021 · doi:10.1145/3447535.3462487 · Cited by 53 · cited 2×
“Chiang and Yin [13] evaluate the impact of tutorials on AR and measure AR through team performance.”
BackgroundAI-Assisted Decision-making: a Cognitive Modeling Approach to Infer Latent Reliance Strategies
Tejeda, H., Kumar, A., Smyth, P., & Steyvers, M. (2022) · Computational Brain & Behavior · doi:10.1007/s42113-022-00157-y · Cited by 53 · cited 2×
“instances where the AI advisor just confirms the human [70].”
BackgroundDo Explanations make VQA Models more Predictable to a Human?Preprint
Chandrasekaran, A., Prabhu, V., Yadav, D., Chattopadhyay, P., & Parikh, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12366 · Cited by 42
Should I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Making
Schemmer, M., Hemmer, P., Kühl, N., Benz, C., & Satzger, G. (2022) · arXiv · doi:10.48550/arxiv.2204.06916 · Cited by 33
Self-citationWhy Does Advice Discounting Occur? The Combined Roles of Confidence and Trust
Wang, X., & Du, X. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.02381 · Cited by 31 · cited 2×
“people show a high level of trust in the advisor, they consider the advice to be high-quality advice from an advisor with good intentions, and they will give more weight to that advice [73].”
BackgroundDoes Explainable Artificial Intelligence Improve Human Decision-Making?
Alufaisan, Y., Marusich, L. R., Bakdash, J. Z., Zhou, Y., & Kantarcioglu, M. (2020) · doi:10.31234/osf.io/d4r9t · Cited by 30
Hemmer, P., Schellhammer, S., Vössing, M., Jakubik, J., & Satzger, G. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/344 · Cited by 29
Self-citationExplaining Explanations to Society
Gilpin, L. H., Testart, C., Fruchter, N., & Adebayo, J. (2019) · arXiv · doi:10.48550/arxiv.1901.06560 · Cited by 28
An Empirical Evaluation of Predicted Outcomes as Explanations in Human-AI Decision-Making
Jakubik, J., Schoeffer, J., Hoge, V., Vössing, M., & Kühl, N. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-23618-1_24 · Cited by 24
Trust and Reliance in XAI -- Distinguishing Between Attitudinal and Behavioral Measures
Scharowski, N., Perrig, S. A. C., von Felten, N., & Brühlmann, F. (2022) · arXiv · doi:10.48550/arxiv.2203.12318 · Cited by 18
How Do Visual Explanations Foster End Users' Appropriate Trust in Machine Learning?Preprint
Yang, F. (2021) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/s9q8t · Cited by 18
Workshop on Trust and Reliance in AI-Human Teams (TRAIT)
Bansal, G., Smith, A., Buçinca, Z., Wu, T., Holstein, K., Hullman, J. R., & Stumpf, S. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3503704 · Cited by 17
Kühl, N., Lobana, J., & Meske, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.08777 · Cited by 16
Selecting Methods for the Analysis of Reliance on Automation
Wang, L., Jamieson, G. A., & Hollands, J. G. (2008) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120805200419 · Cited by 13
AI-Assisted Decision-Making: A Cognitive Modeling Approach to Infer Latent Reliance StrategiesPreprint
Tejeda, H., Kumar, A., Smyth, P., & Steyvers, M. (2022) · doi:10.31234/osf.io/q6yz4 · Cited by 10
On the Effect of Information Asymmetry in Human-AI Teams
Hemmer, P., Schemmer, M., Kühl, N., Vössing, M., & Satzger, G. (2022) · arXiv · doi:10.48550/arxiv.2205.01467 · Cited by 9
Self-citationVisual correspondence-based explanations improve AI robustness and human-AI team accuracyPreprint
Nguyen, G., Taesiri, M. R., & Nguyen, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.00780 · Cited by 8
Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA
Gonzalez, A. V., Bansal, G., Fan, A., Jia, R., Mehdad, Y., & Iyer, S. (2020) · arXiv · doi:10.48550/arxiv.2012.15075 · Cited by 7
Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-MakingPreprint
Schoeffer, J., De‐Arteaga, M., & Kuehl, N. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.11812 · Cited by 3
Self-citationAlnowami, M. R., Abolaban, F. A., Hijazi, H., et al. (2022) · Applied Sciences · doi:10.3390/app12020725 · Cited by 2 · cited 1×
“In the past, research has predominantly focused on maximizing the reliance [35], trust [66], utilization [2], compliance [37], or acceptance [65] of AI advice [34].”
BackgroundSelecting Methods for the Analysis of Reliance on Automation: (578112012-020)
Wang, L., Jamieson, G. A., & Hollands, J. G. (2008) · PsycEXTRA Dataset · doi:10.1037/e578112012-020 · Cited by 2
Explainable AI for Interpretable Credit ScoringPreprint
Demajo, L. M., Vella, V., & Dingli, A. (2020) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2012.03749
On the influence of explainable AI on automation bias
Schemmer, M., Kühl, N., Benz, C., & Satzger, G. (2022) · arXiv · doi:10.48550/arxiv.2204.08859
Self-citation
Research works printed but not linked: 3
- [29] Patrick Hemmer, Max Schemmer, Michael Vössing, and Niklas Kühl. 2021. Human-AI Complementarity in Hybrid Intelligence Systems: A Structured Literature Review. PACIS 2021 Proceedings (2021). -
- [66] Keng Siau and Weiyu Wang. 2018. Building trust in artificial intelligence, machine learning, and robotics. Cutter business technology journal 31, 2 (2018), 47-53. -
- [68] Philipp Spitzer, Niklas Kühl, and Marc Goutier. 2022. Training Novices: The Role of Human-AI Collaboration and Knowledge Transfer. Workshop on Human-Machine Collaboration and Teaming (HM-CaT 2022). The 39th International Conference on Machine Learning, Baltimore, Maryland, USA (2022). -
Cited by
Gomez, C., Cho, S. M., Ke, S., Huang, C.-M., & Unberath, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1521066 · Cited by 209
Enhancing AI-Assisted Group Decision Making through LLM-Powered Devil's Advocate
Chiang, C.-W., Lu, Z., Li, Z., & Yin, M. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645199 · Cited by 142
“For example, many studies specifically examine whether people exhibit any over-reliance or under-reliance on AI recommendations [11, 15, 23, 33, 37, 45, 53, 56, 72, 76, 84, 98, 99, 105, 109, 110].”
BackgroundMehrotra, S., Degachi, C., Vereschak, O., Jonker, C. M., & Tielman, M. L. (2024) · ACM Journal on Responsible Computing · doi:10.1145/3696449 · Cited by 144
Kim, S. S. Y., Liao, Q. V., Vorvoreanu, M., Ballard, S., & Vaughan, J. W. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3658941 · Cited by 147
Explainable AI improves task performance in human–AI collaboration
Senoner, J., Schallmoser, S., Kratzwald, B., Feuerriegel, S., & Netland, T. H. (2024) · Scientific Reports · doi:10.1038/s41598-024-82501-9 · Cited by 202
“Unlike previous works on studying task performance in human–AI collaboration 41–44 , we (i) conducted experiments of two real-world job tasks in manufacturing and medicine and (ii) recruited domain experts for those tasks, i.e., factory workers and radiologists.”
Metadata
- Authors
- Schemmer, M., Kuehl, N., Benz, C., Bartos, A., & Satzger, G.
- Affiliations
- Karlsruhe Institute of Technology (DE)
- Venue
- Proceedings of the 28th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2023-03-27
- Pages
- 410-422
- OpenAlex
- W4360991389
- Semantic Scholar
- 95f751e8535a6d7f8fe733ab4729f738ef0566c4
- Cited by
- 208
- References
- 43
- Citations by year
- 2026: 93 · 2025: 71 · 2024: 37 · 2023: 6
- FWCI
- 36.59
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- Repository KITopen (Karlsruhe Institute of Technology) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- KITopen
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- AI advice, AI explanations, human-AI decision-making, trust in AI, measurement concepts, investment decision-making, treatment decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.