CoChat
Sign Up
Cited by 58Human-Automation Interaction and SafetyDecision-Making and Behavioral EconomicsEthics and Social Impacts of AI

The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index; 39 of the 53 sources its claims cite were checked for support.

A Decision Theoretic Framework for Measuring AI Reliance

Open access
Guo, Ziyang, Wu, Yifan, Hartline, Jason D., Hullman, Jessica·2024·The 2024 ACM Conference on Fairness, Accountability, and Transparency
DOI 10.1145/3630106.3658901
Open this paper in CoChat — free account

Abstract

Humans frequently make decisions with the aid of artificially intelligent (AI) systems. A common pattern is for the AI to recommend an action to the human who retains control over the final decision. Researchers have identified ensuring that a human has appropriate reliance on an AI as a critical component of achieving complementary performance. We argue that the current definition of appropriate reliance used in such research lacks formal statistical grounding and can lead to contradictions. We propose a formal definition of reliance, based on statistical decision theory, which separates the concepts of reliance as the probability the decision-maker follows the AI’s recommendation from challenges a human may face in differentiating the signals and forming accurate beliefs about the situation. Our definition gives rise to a framework that can be used to guide the design and interpretation of studies on human-AI complementarity and reliance. Using recent AI-advised decision making studies from literature, we demonstrate how our framework can be used to separate the loss due to mis-reliance from the loss due to not accurately differentiating the signals. We evaluate these losses by comparing to a baseline and a benchmark for complementary performance defined by the expected payoff achieved by a rational decision-maker facing the same decision task as the behavioral decision-makers.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes a decision-theoretic definition of AI reliance that separates the probability of following AI advice from the belief-updating and discrimination challenge, with rational baseline and benchmark for complementary performance. It reanalyzes three prior AI-advised decision-making experiments to decompose behavioral loss into reliance loss and discrimination loss.

Key findings

  • The conventional definition of appropriate reliance—submitting the AI recommendation when correct and rejecting it when incorrect—lacks formal statistical grounding and can produce contradictions; the authors define reliance as \(\gamma = \Pr[a = y_{AI} \mid y_{AI} \neq y_H]\) and appropriate reliance as the rational decision-maker’s reliance level \(\gamma^r\).
  • The framework decomposes the gap between best attainable performance \(R\) and observed behavior \(B\) into reliance loss \((R - R_m)/\Delta\) and discrimination loss \((R_m - B)/\Delta\), where \(R_m\) is the mis-reliant rational benchmark and \(\Delta = R - R_\emptyset\) is the value of rational complementation.
  • In Lai and Tan [21], the rational baseline dominates almost all other quantities except the rational benchmark across explanation conditions, and the rational benchmark only marginally improves over the baseline, suggesting the design was poorly suited for studying complementary performance; heatmap and accuracy conditions show similar reliance loss, but accuracy has smaller discrimination loss.
  • In Bansal et al. [3], behavioral performance surpasses the rational baseline in all explanation conditions, but remains far below the rational benchmark; reliance loss is minor while discrimination loss is the main source of loss, and NHST at \(\alpha = 0.05\) shows no significant explanation improvements.
  • In Fogliato et al. [12], behavioral performance is higher for the probability task than the binary decision task; non-anchoring shows better relative performance and lower reliance loss than anchoring, but slightly higher discrimination loss, contradicting the original similarity conclusion.

Methods

  • The paper formalizes AI-advised decision-making as a derived binary-adoption task with payoff-related state \(\hat{\theta} = (y, y_H, y_{AI})\), action \(\hat{a} \in \{0 = \text{human}, 1 = \text{AI}\}\), signal \(\hat{v} = \{x, y_H, y_{AI}\}\), and scoring rule; it defines rational baseline \(R_\emptyset\), rational benchmark \(R\), mis-reliant rational benchmark \(R_m\), behavioral performance \(B\), and value of rational complementation \(\Delta\).
  • For infinite signal spaces, the rational benchmark and \(R_m\) are approximated by an upper bound using the empirical distribution and a lower bound using K-means discretization with cross-validation; bootstrap is used to quantify uncertainty.
  • The framework is demonstrated by reanalyzing three published experiments: Lai and Tan [21] on deception detection, Bansal et al. [3] on LSAT questions, and Fogliato et al. [12] on recidivism risk, using the original authors’ data.

Limitations

  • Stated: the framework formalizes AI-advised decision-making as a binary choice between a human recommendation and an AI recommendation, which may not suit continuous recommendation spaces such as regression.
  • Stated: only two losses are identified—reliance loss and discrimination loss—though more fine-grained losses may exist; for example, discrimination loss can arise from misidentifying the AI’s correctness or the human’s correctness.
  • Stated: the framework assumes the human has an inner model and consults the AI recommendation; if the human ignores the AI or anchors on it without forming their own prediction, reliance measurement becomes inaccurate. Evident from the text: the demonstrations are reanalyses of three prior studies rather than new user experiments, and infinite signal spaces require upper- and lower-bound approximations.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3630106.3658901

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

39 of 53 cited sources checked · 16 supported · 1 flagged · accuracy not yet measured

  • Premisecitation
    § 6 DISCUSSION · p. 16

    Prior work has evaluated reliance in action space only.

    “in contrast to prior work which has evaluated reliance in action space only [3, 28, 35].”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [3] [3] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (, Yokohama, Japan,) (CHI '21) . Association for Computing Machinery, New York, NY, USA, Article 81, 16 pages. https://doi.org/10.1145/3411764.3445717 - source

      The paper evaluates reliance through decisions, but not only through actions. In §5.3 (p. 11), it also codes participants’ self-reported collaboration approaches, including following the AI, using it as a prior guide, and checking decisions against it; participants also reported when they considered the AI trustworthy. (LLM, conf=0.84)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [28] [28] Max Schemmer, Niklas Kuehl, Carina Benz, Andrea Bartos, and Gerhard Satzger. 2023. Appropriate reliance on AI advice: Conceptualization and the effect of explanations. In Proceedings of the 28th International Conference on Intelligent User Interfaces . 410-422. - source
    • cites [35] [35] Fumeng Yang, Zhuanyi Huang, Jean Scholtz, and Dustin L Arendt. 2020. How do visual explanations foster end users' appropriate trust in machine learning?. In Proceedings of the 25th international conference on intelligent user interfaces . 189-201. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data
    § 5.1 On Human Prediction with Explanations and Predictions of Machine Learning Models [21] · p. 9

    In Lai and Tan [21], the rational benchmark only improves marginally over the rational baseline.

    “the rational benchmark only improves marginally over the rational baseline”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [21] [21] Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency . 29-38. - source

      Section 4.1 (p. 6) reports that explanations alone slightly improve human accuracy over the control condition. But our copy does not identify a “rational benchmark” or “rational baseline,” so that result cannot establish the claimed comparison. Searches for both terms found no matching comparison. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
    • The authors state this as an assumption: The empirical distribution (e.g., the 4000 observations) can be treated as the true data generating model for approximating the rational benchmarks and the mis-reliant rational benchmark.
  • Findingcitation
    § 5.2 Does the Whole Exceed its Parts? [3] · p. 12

    In Bansal et al. [3], behavioral decision-makers have substantially lower performance compared to the rational benchmark due to large discrimination loss.

    “the behavioral decision-makers have substantially lower performance compared to the rational benchmark due to large discrimination loss”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [3] [3] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (, Yokohama, Japan,) (CHI '21) . Association for Computing Machinery, New York, NY, USA, Article 81, 16 pages. https://doi.org/10.1145/3411764.3445717 - source

      The Background and Related Work (p. 3) reports that some human–AI teams performed worse than AI acting alone, but does not compare behavioral decision-makers with a rational benchmark. Searches for “discrimination loss” and “rational benchmark” found no passage attributing such a performance gap to discrimination loss. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • The authors state this as an assumption: The empirical distribution (e.g., the 4000 observations) can be treated as the true data generating model for approximating the rational benchmarks and the mis-reliant rational benchmark.
  • Interpretationcitation
    § 5.1 On Human Prediction with Explanations and Predictions of Machine Learning Models [21] · p. 9

    In Lai and Tan [21], the rational decision-maker does not gain much from access to human recommendations.

    “i.e., the rational decision-maker does not gain much from access to human recommendations”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [21] [21] Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency . 29-38. - source

      The Experimental Setup (pp. 3–4) studies humans receiving assistance from machine-learning models, not a rational decision-maker receiving human recommendations. The Results (p. 5) assess the effect of machine predictions on human performance. Searches for “rational decision maker” and “human recommendations” found no passage addressing the claimed benefit of access to human recommendations. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • The authors state this as an assumption: The empirical distribution (e.g., the 4000 observations) can be treated as the true data generating model for approximating the rational benchmarks and the mis-reliant rational benchmark.
  • Premisecitation
    § 1 INTRODUCTION · p. 1

    Human-AI teams under-perform the AI alone in many tasks.

    “However, many studies have shown that human-AI teams under-perform the AI alone in tasks [3, 4, 6, 14, 18, 20, 21, 30].”
    Quote in the paperSubstantial claim5 of 8 sources checked
    • cites [3] [3] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (, Yokohama, Japan,) (CHI '21) . Association for Computing Machinery, New York, NY, USA, Article 81, 16 pages. https://doi.org/10.1145/3411764.3445717 - source
    • cites [4] [4] Zana Buçinca, Phoebe Lin, Krzysztof Z. Gajos, and Elena L. Glassman. 2020. Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI '20) . Association for Computing Machinery, New York, NY, USA, 454-464. https://doi.org/10.1145/3377325.3377498 - source
    • cites [6] [6] Adrian Bussone, Simone Stumpf, and Dympna O'Sullivan. 2015. The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems. In 2015 International Conference on Healthcare Informatics . 160-169. https://doi.org/10.1109/ICHI.2015.26 - source

      In “Why Explanations (RQ2)” (p. 7), the paper reports that participants made wrong decisions in over half the instances, supporting the narrower point that people using the decision-support system can perform poorly. But “Prototypes” (p. 5) says the diagnoses were mocked up, and the paper does not establish that human–AI teams underperformed an AI acting alone across many tasks. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [14] [14] Ben Green and Yiling Chen. 2019. The principles and limits of algorithm-in-the-loop decision making. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1-24. - source
    • cites [18] [18] Maia Jacobs, Melanie F Pradier, Thomas H McCoy Jr, Roy H Perlis, Finale Doshi-Velez, and Krzysztof Z Gajos. 2021. How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selection. Translational psychiatry 11, 1 (2021), 108. - source
    • cites [20] [20] Igor Kononenko. 2001. Machine learning for medical diagnosis: history, state of the art and perspective. Artificial Intelligence in medicine 23, 1 (2001), 89-109. - source
    • cites [21] [21] Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency . 29-38. - source
    • cites [30] [30] Michelle Vaccaro and Jim Waldo. 2019. The effects of mixing machine learning and human judgment. Commun. ACM 62, 11 (2019), 104-110. -

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • It may state two separate claims as one. 14
  • Probably not a substantial claim. 34

  • It may state two separate claims as one.

    Findingown data
    § 5.2 Does the Whole Exceed its Parts? [3] · p. 12

    In Bansal et al. [3], there are no significant reductions in either discrimination loss or reliance loss.

    “Using our approach we confirm there are not significant reductions in either discrimination loss or reliance loss.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [3] [3] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (, Yokohama, Japan,) (CHI '21) . Association for Computing Machinery, New York, NY, USA, Article 81, 16 pages. https://doi.org/10.1145/3411764.3445717 - source

      Section 5.1 (pp. 7–8) reports no significant improvement in team performance from explanations over confidence scores, which supports the broad finding of no significant improvement. But our copy does not report tests or reductions for the specific measures “discrimination loss” and “reliance loss”; searches for both terms found no such results. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • The authors state this as an assumption: The empirical distribution (e.g., the 4000 observations) can be treated as the true data generating model for approximating the rational benchmarks and the mis-reliant rational benchmark.

  • Probably not a substantial claim.

    Premisecitation
    § 1 INTRODUCTION · p. 1

    AI-advised decision making, in which a human has access to the AI recommendation and can choose whether to follow it, is often preferred as a means of retaining human control.

    “AI-advised decision making, in which a human decision-maker has access to the recommendation of an artificial intelligence (AI system) and can choose whether or not to follow it, is often preferred as a means of retaining human control [3] in deploying predictive models.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [3] [3] Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (, Yokohama, Japan,) (CHI '21) . Association for Computing Machinery, New York, NY, USA, Article 81, 16 pages. https://doi.org/10.1145/3411764.3445717 - source

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductcitation
    § B FORMALIZED DECISION TASKS · p. 23

    The feature x is included in the binary-adoption signals in Table 4 because yH and yAI in Lai and Tan [21] are both binary-valued, which makes the signal space too small to generate a meaningful benchmark.

    “Note that the feature 𝑥 is included in the binary-adoption signals in Table 4 because 𝑦 𝐻 and 𝑦 𝐴𝐼 in Lai and Tan [21] are both binary-valued, which makes the signal space too small to generate a meaningful benchmark.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Lai and Tan [21] [21] Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency . 29-38. - source

  • It may state two separate claims as one.

    Interpretationcitation
    § A. Quantile Plot of the Expected Payoff of Al Prediction and Human Prediction · p. 15

    Letting behavioral decision-makers make a decision by themselves first can improve their reliance, but not necessarily help them distinguish between signals where the AI recommendation is expected to outperform the human recommendation and signals where the human recommendation is expected to outperform the AI recommendation.

    “This suggests that letting the behavioral decision-makers make a decision by themselves first (a.k.a., the non-anchoring effects) can improve their reliance, but not necessarily help them distinguish between the signals where the AI recommendation is expected to outperform the human recommendation and the signals where the human recommendation is expected to outperform the AI recommendation.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [12] [12] Riccardo Fogliato, Alexandra Chouldechova, and Zachary Lipton. 2021. The impact of algorithmic risk assessments on human predictions and its analysis via crowdsourcing studies. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1-24. - source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • The authors state this as an assumption: The empirical distribution (e.g., the 4000 observations) can be treated as the true data generating model for approximating the rational benchmarks and the mis-reliant rational benchmark.

  • It may state two separate claims as one.

    Findingcitation
    § 5.1 On Human Prediction with Explanations and Predictions of Machine Learning Models [21] · p. 9

    In Lai and Tan [21], the heatmap explanation condition and the accuracy condition show similar reliance loss, but the discrimination loss in the accuracy condition is smaller.

    “the condition with heatmap explanations and the condition directly providing model accuracy show similar reliance loss (Figure 2C) but the discrimination loss in the latter is smaller than the former.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [21] [21] Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency . 29-38. - source
    • The authors state this as an assumption: The empirical distribution (e.g., the 4000 observations) can be treated as the true data generating model for approximating the rational benchmarks and the mis-reliant rational benchmark.

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Interpretationnone given
    § 3.1 Rational Decision-Maker · p. 5

    The value of rational complementation provides a scale for comparing expected performance in terms of the 'lift' from having access to the information in the signals.

    “The value of rational complementation provides a scale for comparing expected performance in terms of the 'lift' we see from having access to the information in the signals.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Interpretationreasoning
    § · Behavioral Performance · p. 6

    R_m must fall between B and R.

    “R m must fall between B and R.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Interpretationnone given
    § 3.1 Rational Decision-Maker · p. 4

    The rational benchmark also describes the appropriate reliance level, which maximizes the expected payoff.

    “The rational benchmark also describes the appropriate reliance level, which maximizes the expected payoff.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may state two separate claims as one.

    Interpretationreasoning
    § · Behavioral Performance · p. 5

    The paper separates the behavioral decision-maker's loss in score into loss from mis-reliance and discrimination loss, and decomposes the difference between the best attainable performance in the study (R) and the observed behavior of study participants (B) into two parts.

    “We separate the behavioral decision-maker's loss in score into two sources: loss from mis-reliance, and what we term discrimination loss, referring to the loss from not accurately distinguishing when the AI recommender has better expected payoff than the human recommender or vice versa.”
    • Also stated in: § · Behavioral Performance · p. 6 · plain
      “In other words, we decompose the difference between the best attainable performance in the study (R) and the observed behavior of study participants (B) into two parts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Interpretationnone given
    § 3.1 Rational Decision-Maker · p. 5

    The appropriate reliance level maximizes the expected score of the decision.

    “Note that the appropriate reliance level maximizes the expected score of the decision.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Interpretationreasoning
    § 3.1 Rational Decision-Maker · p. 5

    The rational benchmark upperbounds the expected performance of any behavioral decision-maker in the experiment.

    “The rational benchmark upperbounds the expected performance of any behavioral decision-maker in the experiment.”
    • Also stated in: § 3.1 Rational Decision-Maker · p. 4 · plain
      “The expected performance of a rational decision-maker who has access to 𝑦 𝐻 , 𝑦 𝐴𝐼 , and 𝑥 is an upperbound on the behavioral decision-maker.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Study design and conductnone given
    § 4 APPLYING THE FRAMEWORK TO AI RELIANCE STUDIES · p. 8

    Estimation of the mis-reliant rational benchmark faces the same risk of overfitting as the rational benchmark.

    “Note that estimation of the mis-reliant rational benchmark faces the same risk of overfitting as the rational benchmark.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.

    Interpretationnone given
    § 3.1 Rational Decision-Maker · p. 5

    In the context of AI-advised decision making, the value of rational complementation represents the maximum improvement of performance we can expect from a complementation of the human and the AI conditioned on the information structure of the signals.

    “In the context of AI-advised decision making, it also represents the maximum improvement of performance we can expect from a complementation of the human and the AI conditioned on the information structure of the signals.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationreasoning
    § · Behavioral Performance · p. 6

    Since R_m and B have the same reliance level and accept the same percentage of AI recommendations, the difference in their decisions lies entirely in accepting the AI recommendations at different instances.

    “Since R m and B have the same reliance level and accept the same percentage of AI recommendations, the difference in the decisions of R m and the decisions of B lies entirely in accepting the AI recommendations at different instances.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Study design and conductown data
    § 4 APPLYING THE FRAMEWORK TO AI RELIANCE STUDIES · p. 8

    The authors bootstrap to compute the expectation and quantify uncertainty of the point estimates.

    “To get a robust estimate, we bootstrap to compute the expectation.”
    • Also stated in: § 4 APPLYING THE FRAMEWORK TO AI RELIANCE STUDIES · p. 8 · plain
      “The estimations of the expected payoff generated through iterations quantify the uncertainty.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Interpretationnone given
    § 1 INTRODUCTION · p. 2

    The framework evaluates the two factors of error within the payoff space by introducing a benchmark for best-attainable performance of human-AI collaboration and a baseline for performance with no collaboration.

    “The framework evaluates these two factors of error within the payoff space (i.e.,how much gain can be achieved if the error is corrected) by introducing a benchmark for the best-attainable performance of human-AI collaboration and a baseline for the performance with no collaboration.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationnone given
    § 4 APPLYING THE FRAMEWORK TO AI RELIANCE STUDIES · p. 8

    When joint behavior is estimated by a model, the quality of behavioral performance estimates depends on how well the model predicts the behavioral data.

    “When we estimate the joint behavior by a model, how good the estimates of behavioral performance are will depend on how well the model predicts the behavioral data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Interpretationcitation
    § 1 INTRODUCTION · p. 2

    The paper's formal definition of AI reliance is grounded in statistical decision theory.

    “Following previous work on generating benchmarks for studies of information displays [34], our approach is grounded in statistical decision theory.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [34] [34] Yifan Wu, Ziyang Guo, Michails Mamakos, Jason Hartline, and Jessica Hullman. 2023. The Rational Agent Benchmark for Data Visualization. arXiv preprint arXiv:2304.03432 (2023). - source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim. It may state two separate claims as one.

    Interpretationreasoning
    § 1 INTRODUCTION · p. 2

    The conventional definition of appropriate reliance is binary, so researchers cannot distinguish whether a human mistakenly used or did not use the AI recommendation when the probability that relying on their own judgment would have been correct is similar to versus very different from the probability that the AI was correct.

    “Another issue with the conventional definition of appropriate reliance is that it is a binary measure. Consequently, researchers cannot distinguish whether the human decision-maker mistakenly used (or did not use) the AI's recommendation in a situation where (A) the probability that relying on their own judgment would have been correct is similar to the probability that the AI was correct versus (”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationreasoning
    § A Decision Theoretic Framework for Measuring AI Reliance · p. 1

    The conventional definition of appropriate reliance used in AI-advised decision-making research lacks formal statistical grounding and can lead to contradictions.

    “We argue that the current definition of appropriate reliance used in such research lacks formal statistical grounding and can lead to contradictions.”
    • Also stated in: § 1 INTRODUCTION · p. 2 · plain
      “We argue that this definition of reliance lacks formal statistical grounding, leading to contradictions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Interpretationnone given
    § · Behavioral Performance · p. 6

    B must be equal to or lower than the rational benchmark R.

    “B must be equal to or lower than the rational benchmark R.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Interpretationreasoning
    § · Behavioral Performance · p. 6

    The mis-reliant rational benchmark R_m represents the best score a decision-maker with a given reliance level γ could attain had they perfectly perceived the probability that the AI is correct relative to the probability that the human is correct on every decision task.

    “Hence, the mis-reliant rational benchmark R m represents the best score an decision-maker with a given reliance level 𝛾 could attain had they perfectly perceived the probability that the AI is correct relative to the probability that the human is correct on every decision task.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may state two separate claims as one.

    Study design and conductreasoning
    § 4 APPLYING THE FRAMEWORK TO AI RELIANCE STUDIES · p. 7

    The authors perform cross-validation on the estimated average payoff to ensure the discretization is not too fine and the estimate does not overfit to the experiment data.

    “We ensure this by performing cross-validation on the estimated average payoff.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Interpretationnone given
    § 1 INTRODUCTION · p. 2

    The paper's definition separates reliance level (the probability the human decision-maker goes with the AI recommendation) from the belief updating that a rational decision-maker is expected to do upon viewing an instance and associated AI recommendation.

    “Our definition separates the concepts of a reliance level (the probability that the human decision-maker goes with the AI recommendation) from the belief updating that a rational decision-maker is expected to do upon viewing an instance and associated AI recommendation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given
    § 3.1 Rational Decision-Maker · p. 4

    Explanations of the AI recommendation may be available to the behavioral decision-maker, but they are a garbling of the feature values and AI recommendation and will not change the rational decision-maker's action.

    “Though some explanations of 𝑦 𝐴𝐼 may also be available to behavioral decision-maker, they are a garbling of 𝑥 and 𝑦 𝐴𝐼 and won't change the rational decision-maker's action.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Study design and conductreasoning
    § 4 APPLYING THE FRAMEWORK TO AI RELIANCE STUDIES · p. 8

    The authors select the number of clusters k to balance the increase in expected payoff and the generalization error.

    “We select 𝑘 to balance the increase in expected payoff and the generalization error.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim.

    Interpretationreasoning
    § 3 DEFINITION OF RELIANCE · p. 4

    The reliance level cannot be defined when the human makes the same recommendation as the AI.

    “The definition targets a conditional probability, because the reliance level cannot be defined when the human makes the same recommendation as the AI.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Interpretationreasoning
    § 1 INTRODUCTION · p. 2

    Without a definition that allows separation of different sources of performance loss, analysis might misinterpret the reasons behind poor experiment results, leading researchers to prioritize less directly relevant follow-up actions.

    “Without a definition that allows separation of different sources of performance loss, the analysis might misinterpret the reasons behind seemingly poor experiment results, leading researchers to prioritize less directly relevant follow-up actions for improving the team.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given
    § · Behavioral Performance · p. 6

    R_m always accepts the top x% AI recommendations ranked by performance advantage over human recommendations, but B may not.

    “R m always accepts the top 𝑥 % AI recommendations ranked by performance advantage over human recommendations, but B may not.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationnone given
    § 6.1 Limitations · p. 17

    In continuous recommendation spaces, the human decision-maker is likely to make a decision different from the human recommendation or the AI recommendation.

    “when the recommendation space is continuous (e.g., regression), the human decision-maker is likely to make a decision that is different from the human recommendation or the AI recommendation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given
    § 6.1 Limitations · p. 17

    Discrimination loss can be caused by misidentifying the probability that the AI is correct or misidentifying the probability that the human is correct.

    “discrimination loss can be caused by two possible reasons: misidentifying the probability that the AI is correct or misidentifying the probability that the human is correct.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may state two separate claims as one.

    Interpretationreasoning
    § 6 DISCUSSION · p. 16

    Without constraints, humans might ignore the AI and just submit the human recommendation or anchor on the AI without thinking to make the decision by themselves, causing inaccurate measurement of reliance.

    “without constraints, they might ignore the AI and just submit the human recommendation or anchor on the AI without thinking to make the decision by themselves. Both of these two cases cause inaccurate measurement of reliance”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 6.1 Limitations · p. 17

    The formalization of AI-advised decision-making as a binary choice may not be suitable for every real world case.

    “However, this may not be suitable for every real world case.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.

    Interpretationreasoning
    § 6 DISCUSSION · p. 16

    The generalizability of the framework to alternative study set-ups holds.

    “the generalizability of our framework to alternative study set-ups still holds.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Interpretationnone given
    § 6 DISCUSSION · p. 16

    Defining a measurement of reliance in payoff space enables the calculation of a benchmark to compare with.

    “Defining a measurement of reliance in payoff space also enables the calculation of a benchmark to compare with”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Interpretationnone given
    § 6 DISCUSSION · p. 16

    Without a well-grounded notion of reliance, studies have limited ability to draw conclusions from a decision-making task on how good the reliance is and whether action should be taken to improve it.

    “Without a well-grounded notion of reliance, such studies have limited ability to draw conclusions from a decision-making task on how good the reliance is and whether action should be taken to improve it.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.

    Interpretationreasoning
    § 6.1 Limitations · p. 17

    Improving the former implies better conveying the AI's accuracy, while improving the latter implies giving information on the human's average performance on the task.

    “Improving the former implies better conveying the AI's accuracy, while improving the latter implies giving information on the human's average performance on the task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Study design and conductnone given
    § C THE RESULTS OF DEMONSTRATIONS USING DISCRETIZED SIGNAL APPROXIMATION · p. 25

    All conclusions analyzed under the results of approximation using discretized signals corroborate with the conclusions in Section 5.

    “All the conclusions analyzed under the results of approximation using discretized signals corroborate with the conclusions we get in Section 5.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationreasoning
    § 6 DISCUSSION · p. 16

    The framework can be applied to situations where the human is both making a recommendation and making the final decision.

    “Our framework can be applied to situations where the human is both making a recommendation and making the final decision, i.e., where the human recommender and decision-maker are the same person.”
    Quote in the paperNot a substantial claim
39 of 39

  • The authors state this as an assumption.

    The human has an inner model of the data-generating process that enables them to make their own prediction about each instance.

    “- (1) The human has an inner model of the data-generating process that enables them to make their own prediction about each instance.”

  • The authors state this as an assumption.

    The human consults the AI recommendation prior to making their decision.

    “- (2) The human consults the AI recommendation prior to making their decision.”

  • The authors state this as an assumption.

    The joint distribution over the payoff-relevant state and the signal is continuous.

    “Assuming continuity on the joint distribution ˜ 𝜋 ( ˆ 𝜃, ˆ 𝑣 ) over the payoff-relevant state ˆ 𝜃 and the signal ˆ 𝑣 , we approximate the rational benchmark by coarsening the signal space into finite discrete signals”

  • The authors state this as an assumption.

    The empirical distribution (e.g., the 4000 observations) can be treated as the true data generating model for approximating the rational benchmarks and the mis-reliant rational benchmark.

    “We use the upper bound (overfit) method to approximate the rational benchmarks and the mis-reliant rational benchmark, i.e., estimating the empirical distribution using the observations of signals and payoff-relevant state and treating the empirical distribution as the true data generating model.”

  • The authors state this as an assumption.

    The rational decision-maker knows the posterior payoffs for each signal.

    “For each signal ˆ 𝑣 𝑖 , the rational decision-maker knows the posterior payoffs”

  • The authors state this as an assumption.

    The rational decision-maker completes the same set of trials as the behavioral decision-makers.

    “we simulate the rational decision-maker completing the same set of trials as the behavioral decision-makers do”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Students' desirable reliance behaviours should be associated with cognitive engagement.

    “Based on previous studies, students' desirable reliance behaviours should be associated with cognitive engagement (Fok & Weld, 2024; Guo et al., 2024; Vasconcelos et al., 2023).”

    From The role of critical thinking on undergraduates' reliance behaviours on generative AI in problem‐solving (2025) · cites this paper as Guo et al., 2024

    The Introduction (p. 2) links a failure to choose appropriately between one’s own decision and the AI’s recommendation to a possible lack of motivation to process information. That supports a proposed connection between reliance and cognitive engagement, but the paper discusses human decision-makers generally, not students, and does not establish an association in students. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Opposite conclusions about which condition has lower reliance loss

    Major

    Section 5.3 says reliance loss is lower without anchoring, whereas Appendix C.3 says anchoring improves reliance loss over non-anchoring.

  • Appendix cites the Bansal figure for Fogliato results

    Minor

    Appendix C.3 cites Figure 8 for Fogliato results, but Figure 8 is captioned as Bansal et al.; the Fogliato results are captioned Figure 10.

  • Appendix cites the Bansal figure for Lai and Tan results

    Minor

    Appendix C.2 cites Figure 8A for a Lai and Tan result, but Figure 8 is captioned as Bansal et al.; the Lai and Tan results are captioned Figure 9.

  • Nine assistance conditions are listed as seven

    Minor

    The sentence says Lai and Tan proposed seven conditions, but the list that follows contains nine distinct conditions.

4 of 4

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. 2018. To trust or not to trust a classifier. Advances in neural information processing systems 31 (2018).

Cited by

Sorted by citations

Related

Metadata

Authors
Guo, Z., Wu, Y., Hartline, J. D., & Hullman, J.
Affiliations
Northwestern University (US)
ORCID
Venue
The 2024 ACM Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2024-06-03
Pages
221-236
DOI
10.1145/3630106.3658901
OpenAlex
W4399363390
Semantic Scholar
66a6b7ed6a53a04cf1ef28a87a1b7867dc7d1329
Cited by
27
References
23
Citations by year
2026: 16 · 2025: 10 · 2024: 1
FWCI
18.10
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
NSF (National Science Foundation) (2211939)
Topics
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
  • Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
AI reliance, human-AI complementarity, statistical decision theory, AI recommendations, human decision-making, signal differentiation, belief formation, expected payoff

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review