CoChat
Sign Up
Cited by 143Human-Automation Interaction and SafetyExplainable Artificial Intelligence (XAI)Ethics and Social Impacts of AI

Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making

Open access
Ma, Shuai, Lei, Ying, Wang, Xinru, Zheng, Chengbo, Shi, Chuhan, Yin, Ming, Ma, Xiaojuan·2023·Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3544548.3581058
Open this paper in CoChat — free accountOpen PDF

Abstract

In AI-assisted decision-making, it is critical for human decision-makers to know when to trust AI and when to trust themselves. However, prior studies calibrated human trust only based on AI confidence indicating AI’s correctness likelihood (CL) but ignored humans’ CL, hindering optimal team decision-making. To mitigate this gap, we proposed to promote humans’ appropriate trust based on the CL of both sides at a task-instance level. We first modeled humans’ CL by approximating their decision-making models and computing their potential performance in similar instances. We demonstrated the feasibility and effectiveness of our model via two preliminary studies. Then, we proposed three CL exploitation strategies to calibrate users’ trust explicitly/implicitly in the AI-assisted decision-making process. Results from a between-subjects experiment (N=293) showed that our CL exploitation strategies promoted more appropriate human trust in AI, compared with only using AI confidence. We further provided practical implications for more human-compatible AI-assisted decision-making.

Integrity signals

Registry

Crossref · 10.1145/3544548.3581058

Cites

73 of 73 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 91 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The three proposed correctness-likelihood exploitation strategies led to more appropriate user trust in AI than baseline conditions, particularly when the AI made incorrect recommendations.

    “our proposed three CL exploitation strategies resulted in more appropriate user trust in AI compared to baseline conditions, especially when the AI gave wrong recommendations.”
    • Also stated in: § Abstract · plain
      “our CL exploitation strategies promoted more appropriate human trust in AI, compared with only using AI confdence.”
    Quote in the paperSubstantial claim
  • Findingown data

    The three proposed correctness-likelihood exploitation strategies improved team performance.

    “The three proposed CL exploitation strategies also led to improved team performance.”
    Quote in the paperSubstantial claim
  • Findingown data

    Qualitative feedback indicated that the three designs prompted participants to rely more on their own thoughts when AI's advice was wrong.

    “Through qualitative feedback from participants, we found that our three designs prompted people to rely more on their own thoughts when AI's advice was wrong (at this time, AI's CL was often lower than human's).”
    Quote in the paperNot a substantial claim
  • Findingown data

    When the human's correctness likelihood was higher, team performance was significantly better than when AI's correctness likelihood was higher (p<.001 in all conditions).

    “when the human CL was higher, the team performance was signifcantly better than when the AI's CL was higher (<.001 in all conditions).”
    Quote in the paperSubstantial claim
  • Findingown data

    In all three proposed conditions, participants agreed with AI significantly more when AI's correctness likelihood was higher than when the human's correctness likelihood was higher (p<.001 in all three).

    “Results showed that all three conditions made people's agreement with AI signifcantly higher when AI's CL was higher than that when the human's CL was higher (<.001 in all three conditions).”
    Quote in the paperSubstantial claim
  • May combine two claims. 48
  • The quote only partly states this claim. 3
  • Classified as probably not a substantial claim. 27

  • May combine two claims.

    Findingown data

    The proposed method for modeling human correctness likelihood was feasible and effective in the two preliminary studies.

    “We demonstrated the feasibility and efectiveness of our model via two preliminary studies.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Obtaining enough decision data to compute human performance on similar task instances, especially for a new task, is often difficult.

  • May combine two claims.

    Findingown data

    Participants' self-reported confidence was poorly calibrated: their actual accuracy and self-reported confidence were statistically unrelated, so self-reported confidence is not a reliable indicator of human correctness likelihood.

    “people usually have poorly-calibrated self-confdence that cannot reliably refect their actual CL [47,[67][68][69]102].”
    • Also stated in: § 3.2.3 Task cases selection. · hedged
      “We carried out a small-scale user study and found that participants had poorly-calibrated subjective confdence.That is, the correlation between their actual accuracy and self-reported confdence is statistically unrelated, suggesting that self-reported confdence is not a reliable human CL indicator.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The proposed method for estimating human correctness likelihood was effective at identifying complementary task instances, outperforming the traditional AI confidence-based method.

    “we validated the efectiveness of our method in identifying complementary task instances (only one in the human-AI team can do it right) compared to the traditional AI confdence-based method.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Obtaining enough decision data to compute human performance on similar task instances, especially for a new task, is often difficult.

  • May combine two claims.

    Findingown data

    The interactive rule set interface was more visually interpretable and better aligned with participants' decision-making processes than the interactive decision tree interface.

    “seven out of the 10 participants using the interactive decision tree interface reported that their actual decision processes could not be well represented by a decision tree.”
    • Also stated in: § 3.3.2 Results. · plain
      “three out of the 10 participants found the decision tree to be visually complex.”
    • Also stated in: § 3.3.2 Results. · plain
      “the interactive rule set interface was considered to be more visually interpretable and more in line with participants' decision-making processes.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Human-AI agreement in AI Confidence was significantly higher than in Adaptive Workflow (p=.017) and Adaptive Recommendation (p<.001), and marginally higher than in Direct Display (p=.07).

    “the human-AI agreement in AI Confdence was signifcantly higher than Adaptive Workfow (=.017), and Adaptive Recommendation (<.001), and it was marginally higher than Direct Display (=.07).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    AI Confidence increased agreement with AI when AI was correct but also increased agreement with AI when AI was wrong.

    “although AI Confdence made humans agree with AI more when AI's recommendation was correct, it also made humans agree with AI's wrong recommendation more.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The three proposed CL exploitation conditions achieved complementary team performance, whereas the AI Confidence condition did not.

    “The proposed three CL exploitation conditions achieved complementary performance while the AI Confdence condition did not.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When the human's correctness likelihood was higher and AI's recommendation was wrong, Direct Display team performance was marginally lower than Adaptive Workflow (p<.1) and significantly lower than Adaptive Recommendation (p<.05).

    “When human's CL was higher & AI's recommendation was wrong, the team performance in Direct Display was marginally lower than Adaptive Workfow (<.1) and signifcantly lower than Adaptive Recommendation (<.05).”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingcitation

    The selected 40 task samples had an AI accuracy of 70%, with balanced positive and negative labels and equal false positive and false negative rates.

    “We then selected 40 task samples over which the AI model had a 70% accuracy with equal positive and negative labels, as well as equal false positive and false positive rates (similar setting as [6]).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [6] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    A similar phenomenon was found in Adaptive Recommendation.

    “Similar phenomenon can be found in Adaptive Recommendation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Team performance in Direct Display, Adaptive Workflow, and Adaptive Recommendation exceeded both AI alone and Human Only.

    “The team performance in Direct Display (=0.758, =0.078), Adaptive Workfow (=0.767, =0.085), Adaptive Recommendation (=0.754, =0.072) surpassed both AI alone (0.7) and Human Only (=0.721, =0.113).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The accuracy improvement from the auto-generated to the human-revised decision-making model was slight and not statistically significant.

    “This shows a slight but not signifcant improvement.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Compared with AI Confidence, participants found the system significantly more complex in Direct Display (p<.001) and Adaptive Workflow (p<.01).

    “compared with AI Confdence, participants found the system signifcantly more complex in Direct Display (<.001) and Adaptive Workfow (<.01)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the Conflict region, team performance in AI Confidence was significantly lower than in Direct Display (p=.002) and Adaptive Workflow (p=.006).

    “in Confict region, the team performance in AI Confdence was signifcantly lower than Direct Display (=.002), Adaptive Workfow (=.006).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In Direct Display, 23% of participants doubted the displayed correctness likelihood and 25% ignored it in decision-making.

    “In Direct Display, 23% of participants doubted the displayed CL, and 25% ignored the CL in the decision-making process.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    No significant differences were observed between AI Confidence and Direct Display in the Low & Wrong region.

    “But no signifcant diferences were observed between AI Confdence and Direct Display.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the Low & Wrong region, team performance in AI Confidence was marginally lower than Adaptive Workflow (p=.088) and significantly lower than Adaptive Recommendation (p<.001).

    “In the Low & Wrong region, the team performance in AI Confdence was marginally signifcantly lower than Adaptive Workfow (=.088) and signifcantly lower than Adaptive Recommendation (<.001).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There were no significant differences among conditions in Trust in AI, Confidence in the decision-making process, Perceived autonomy, Satisfaction, Future use, Trust in the estimation of human-AI correctness likelihood, and Perceived usefulness of that estimation.

    “in terms of Trust in AI, Confdence in the decisionmaking process, Perceived autonomy, Satisfaction, Future use, Trust in the estimation of human-AI CL, and Perceived usefulness of estimation of human-AI CL, there is no signifcant diference among diferent conditions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Participants in Adaptive Workflow found the estimated human-AI correctness likelihood marginally more helpful than in Direct Display (p=.056) and Adaptive Recommendation (p=.073).

    “participants in Adaptive Workfow found the estimated human-AI CL marginally more helpful than Direct Display (=.056) and Adaptive Recommendation (=.073).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the High & Wrong region, team performance in AI Confidence was significantly lower than Direct Display (p=.005) and Adaptive Workflow (p=.022), and marginally lower than Adaptive Recommendation (p=.056).

    “team performance in AI Confdence was signifcantly lower than Direct Display (=.005), Adaptive Workfow (=.022) and marginally lower than Adaptive Recommendation (=.056).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    A trend was observed that Adaptive Workflow led to high mental demand for participants.

    “we observe a trend that Adaptive Workfow leads to high mental demand for participants.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The three proposed communication strategies achieved complementary performance, exceeding AI Only and Human Only, but the improvement was not remarkable (about 3-4%).

    “Our study found that although the team performance in the proposed three conditions exceeded AI Only and Human Only, the improvement was not”
    • Also stated in: § 7 CONCLUSION · plain
      “The proposed three communication strategies achieve complementary performance compared with AI accuracy (0.7) and Human Only (0.72).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In Direct Display, 74% of participants referred to correctness likelihood; in Adaptive Workflow and Adaptive Recommendation, 53% and 62% were affected by the adaptive process, respectively.

    “In Direct Display, 74% of the participants referred to CL, and in Adaptive Workfow and Adaptive Recommendation, 53% and 62% of participants were afected by the adaptive process respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In Adaptive Recommendation, 62% of participants would think independently when they could not see an AI recommendation, and 33% said they always thought on their own.

    “62% of participants in Adaptive Recommendation would think independently when they could not see an AI recommendation, and 33% said they always thought on their own.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the Consistent region, team performance in AI Confidence was marginally lower than Adaptive Workflow (p=.074) and significantly lower than Adaptive Recommendation (p<.001).

    “the team performance in AI Confdence was marginally signifcantly lower than Adaptive Workfow (=.074) and signifcantly lower than Adaptive Recommendation (<.001).”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingnone given

    No significant difference was found between AI Confidence and Adaptive Recommendation in the Conflict region.

    “No signifcant diference was found between AI Confdence and Adaptive Recommendation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Some participants did not think their ability could be easily and reliably estimated by the system, which hindered the potential of the method.

    “Participants' open-ended feedback suggests that some participants did not think that their ability could be easily and reliably estimated by the system, which hindered the potential of our method.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The three CL exploitation mechanisms improved people's appropriate trust in AI, but some participants still held onto their misjudgments.

    “Although the three CL exploitation mechanisms proposed in this paper improved people's appropriate trust in AI, we found that there were still some participants who held onto their misjudgments.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    The proposed modeling method can estimate people's predictions in addition to approximating their correctness likelihood.

    “In addition to approximating people's CL, our proposed modeling method is able to estimate people's predictions.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In Adaptive Recommendation, participants seemed to form a pattern of skepticism of AI because they often could not see AI suggestions, which might hinder their utilization of AI's assistance when AI's correct advice is shown.

    “For example, in Adaptive Recommendation, we found that the participants seemed to form a pattern of skepticism of AI because they often could not see AI suggestions, which might hinder their utilization of AI's assistance when AI's correct advice is shown.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisereasoning

    Prior work on trust calibration in AI-assisted decision-making relied solely on AI confidence as an indicator of AI correctness likelihood and did not account for the human's correctness likelihood, which limits team decision-making.

    “prior studies calibrated human trust only based on AI confdence indicating AI's correctness likelihood (CL) but ignored humans' CL, hindering optimal team decision-making.”
    • Also stated in: § 1 INTRODUCTION · plain
      “these methods try to steer how much humans value AI's suggestions solely based on AI's correctness likelihood (illustrated in Figure 1 (a)) while largely overlooking humans' correctness likelihood in each case.”
    • Also stated in: § 2 RELATED WORK 2.1 Trust Calibration in AI-Assisted Decision-Making · plain
      “these approaches calibrate humans' trust only based on AI's CL and ignore human CL.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [6] source

      Not checked in the free run.

    • cites [82]

      Not checked in the free run.

    • cites [106] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    After attention-check filtering, 289 valid responses remained, distributed as Human Only 59, AI Confidence 59, Direct Display 59, Adaptive Workflow 56, and Adaptive Recommendation 56.

    “After fltering based on the attention-check questions, we obtain 289 valid responses (Human Only: 59, AI Confdence: 59, Direct Display: 59, Adaptive Workfow: 56, Adaptive Recommendation: 56).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Previous work has shown that displaying AI confidence does not necessarily improve team performance.

    “This fnding is consistent with previous work revealing that showing AI confdence does not necessarily improve team performance [82,106].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [82]

      Not checked in the free run.

    • cites [106] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The final qualitative themes were discussed and harmonized over several iterations.

    “The fnal themes were discussed and harmonized over several iterations”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    When AI's correctness likelihood was higher, people sometimes followed their own wrong judgments and under-trusted AI because their capability was not as good as AI's.

    “The possible reason was that when AI's CL was higher, since human's capability was not as good as AI's, people sometimes followed their own wrong judgments and thus under-trusted AI.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The lack of difference when AI was correct may be because the task instances where humans and AI could both be correct were highly overlapped.

    “We infer this might be because the task instances where humans and AI could make correct predictions were highly overlapped.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Human Only performance reached 80% when AI gave correct advice, indicating weak human-AI complementarity and limited room for improvement in those cases.

    “It can be seen from Human Only that in the case where AI gave correct advice, even if people did not get any assistance from the AI, their performance also reached 80% (agree with AI on 80% cases), which indicates the complementarity of human and AI in such situations was relatively weak, and the room for improvement was thus limited.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study was approved by the University IRB.

    “The study is approved by the University IRB.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The study recruited 300 participants, with 60 participants per condition.

    “We recruited 300 participants (60 for each condition) from 1 Prolifc .”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Research on cognitive forcing has explored methods that push human decision-makers to spend more time deliberating about problems.

    “research on”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [14] source

      Not checked in the free run.

    • cites [78] source

      Not checked in the free run.

    • cites [82]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Certain machine learning classifiers cannot directly output calibrated confidence scores, making post-hoc calibration necessary.

    “some ML classifers (such as SVM and neural networks) cannot directly generate calibrated confdence scores [6,41,106], so post-hoc calibration is required (such as Platt Scaling or Isotonic Regression [41,80]).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [6] source

      Not checked in the free run.

    • cites [41]

      Not checked in the free run.

    • cites [106] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    For qualitative analysis, two authors coded open-ended feedback using inductive thematic analysis.

    “For qualitative analysis, two authors coded the open-ended feedback via inductive thematic analysis [45].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45]

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    In Adaptive Recommendation, when the human's correctness likelihood was higher, AI suggestions were not shown, so participants could not benefit from AI's correct suggestions.

    “This might be because in Adaptive Recommendation condition, when the human's CL was higher, the AI's suggestions were not shown, thus participants could not get the help of the AI's correct suggestions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The study obtained 293 complete submissions.

    “In total, we got 293 complete submissions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Although the human-AI correctness likelihood method outperformed the AI confidence-based method in identifying complementary cases, the improvement was not very large.

    “although our method is better than the AI confdence-based method in identifying complementary cases, the improvement is not very large.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    People generally tend to use System 1 thinking, which gives rise to various cognitive biases.

    “as people are generally inclined to engage in System 1 thinking [47], there are often various cognitive biases”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [47]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Rule-based methods are advantageous because they are interpretable.

    “rule-based methods have the advantage of interpretability [2,51,55,60,61].”
    Quote in the paperNot a substantial claim0 of 5 sources checked
    • cites [2] source

      Not checked in the free run.

    • cites [51] source

      Not checked in the free run.

    • cites [55] source

      Not checked in the free run.

    • cites [60]

      Not checked in the free run.

    • cites [61] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The interfaces were tested in a pilot study to ensure that participants could follow the workflow clearly.

    “The interfaces were tested through a pilot study to ensure that the workfow was clear for participants to follow.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The income prediction task is suitable for this study because it demands little domain expertise and carries limited risk, making it appropriate for non-expert participants.

    “This task is suitable for our study since it requires little domain expertise and imposes relatively limited risks, and thus is amenable for non-expert participants [39].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [39] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Two attention-check questions were inserted at the end of the tutorial to filter out participants who did not read the introduction carefully.

    “We inserted two attentioncheck questions at the end of the tutorial to help flter out participants who did not read the introduction carefully.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Automatically adapting the decision-making interface can counter possible human cognitive biases and avoid making people directly deal with probabilities.

    “This can counter possible human cognitive biases [8,97] and avoid making people directly deal with probabilities.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [8] source

      Not checked in the free run.

    • cites [97] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    This doubt and ignoring may explain why human-AI agreement did not reach 100% even when AI's correctness likelihood was higher, and why team performance did not reach 100% when AI was correct and had higher correctness likelihood.

    “This also echoes the results that even though the AI's CL was higher, the human-AI agreement still did not reach 100% (Figure 6), and it might explain why when AI's CL was higher and AI's recommendation was correct, the team performance still did not reach 100% accurate (Figure 8 (b)).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The study shows promise in modeling humans' correctness likelihood at the task-instance level and using human and AI correctness likelihoods to promote appropriate human trust in AI-assisted decision-making.

    “our study shows the promise of modeling humans' correctness likelihood (CL) at a task instance level and leveraging human and AI CLs to promote appropriate human trust in AI-assisted decision-making.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The human correctness likelihood modeling and communication method proposed in the paper is an initial step toward that goal.

    “The human CL modeling and communication method proposed in this paper is an initial step toward this goal.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Overall, the three proposed CL exploitation methods promoted appropriate trust in AI, especially by reducing over-trust without increasing under-trust, and thereby led to better team performance.

    “Overall, our proposed three CL exploitation methods promoted humans' appropriate trust in AI (especially reducing humans' over-trust and without increasing under-trust) and thus led to better team performance.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: When AI is wrong, lower agreement with AI is better.

  • May combine two claims.

    Interpretationown data

    This is consistent with the results that the three proposed methods affected people's agreement with AI and promoted appropriate trust.

    “This is consistent with the results that the three proposed methods could efectively afect people's agreement with AI (Figure 6), and promote humans' appropriate trust (Figure 7).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: When AI is wrong, lower agreement with AI is better.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    In Adaptive Recommendation, people may have developed independent thinking without relying on AI advice because they often could not see it, which might lead to under-trust.

    “probably because, in Adaptive Recommendation condition, people developed a mode of independent thinking without relying on AI advice because they often could not see AI advice, which might lead to under-trust.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    This reflects that participants formed a pattern of not relying on AI, so human-AI agreement was lowest when AI's correctness likelihood was higher.

    “This refects that participants formed a pattern of not relying on the AI, so the human-AI agreement is lowest when the AI's CL is higher (Figure 6).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Previous work suggests that people, especially those not good at mathematical thinking, lack the ability to act on numbers such as confidence and accuracy, and easily misinterpret what a probability value means.

    “previous works suggest that people, especially those who are not good at applying mathematical thinking, lack the ability to act on numbers (e.g., confdence, accuracy) [9,13,54,92] and easily interpret what a probability value actually means [21,79,85].”
    Quote in the paperSubstantial claim0 of 7 sources checked
    • cites [9]

      Not checked in the free run.

    • cites [13] source

      Not checked in the free run.

    • cites [54] source

      Not checked in the free run.

    • cites [92]

      Not checked in the free run.

    • cites [21]

      Not checked in the free run.

    • cites [79]

      Not checked in the free run.

    • cites [85]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Individuals' confidence in their own capabilities may mismatch with their actual capabilities, leading to overconfidence or underconfidence for both experts and lay people.

    “individuals' confdence in their own capabilities may mismatch with their actual capabilities, leading to overconfdence or underconfdence for both experts and lay people [67-69, 95, 102].”
    Quote in the paperSubstantial claim0 of 5 sources checked
    • cites [67]

      Not checked in the free run.

    • cites [68]

      Not checked in the free run.

    • cites [69]

      Not checked in the free run.

    • cites [95]

      Not checked in the free run.

    • cites [102]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Some studies found that the framing of confidence may affect people's perception of risk.

    “In addition, some studies found that the framing of confdence may afect people's perception of risk [20].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Previous works show that the availability of AI's explanations, regardless of their correctness, is likely to increase people's trust in AI.

    “Previous works show that the availability of AI's explanations, regardless of their correctness, is likely to increase people's trust in AI [6,54,81,100].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [6] source

      Not checked in the free run.

    • cites [54] source

      Not checked in the free run.

    • cites [81] source

      Not checked in the free run.

    • cites [100] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Rules only provide a general model and cannot cover all edge cases.

    “However, rules only provide a general model and cannot cover all edge cases.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The rule-based approach used to help users understand and modify the auto-generated decision model may not be suitable for more complex decision tasks such as those involving text or image data.

    “However, this approach may not be suitable for more complex decision tasks such as those involving text or image data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    The estimation of human CL can be imperfect, and an AI model's confidence can sometimes be poorly calibrated.

    “Second, our estimation of human CL can be imperfect, and an AI model's confdence can sometimes be poorly-calibrated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recent work highlights the dual process of cognition when people process information in decision-making.

    “Besides, recent work highlights the dual process of cognition when people process information in decision-making [13,17,47,101].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [13] source

      Not checked in the free run.

    • cites [17] source

      Not checked in the free run.

    • cites [47]

      Not checked in the free run.

    • cites [101]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In real-world decision-making, users' decision-making models can change as users interact with AI services and encounter more task instances, so a static model is not enough.

    “However, in real-world decision-making, users' decision-making models can change as users interact with AI services and encounter more task instances [64], so a static model is not enough.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [64] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    The proposed framework to calibrate humans' trust based on both sides' capabilities can be generalized to different AI-assisted decision-making scenarios where collaboration is needed.

    “Nevertheless, we believe our proposed framework to calibrate humans' trust based on both sides' capabilities can be generalized to diferent AI-assisted decision-making scenarios where collaboration is needed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationown data

    To ensure complementary performance, besides calibrating people's trust in AI, it is necessary to harness the complementarity of human and AI intelligence.

    “Therefore, echoing [6,106], to ensure complementary performance, besides calibrating people's trust in AI, it is necessary to harness the complementarity of human and AI intelligence [5,103], perhaps by training an AI that can complement humans' knowledge and error regions [3].”
    Quote in the paperNot a substantial claim0 of 5 sources checked
    • cites [6] source

      Not checked in the free run.

    • cites [106] source

      Not checked in the free run.

    • cites [5] source

      Not checked in the free run.

    • cites [103] source

      Not checked in the free run.

    • cites [3]

      Not checked in the free run.

  • May combine two claims.

    Study design and conductnone given

    The study was conducted on non-expert users in low-stake decision-making tasks.

    “Second, our study was conducted on non-expert users in low-stake decision-making tasks.”
    Quote in the paperSubstantial claim
70 of 70

  • The paper relies on this without support.

    Obtaining enough decision data to compute human performance on similar task instances, especially for a new task, is often difficult.

    “However, it is often difcult to obtain enough decision data to compute human performance on similar tasks.”
    “However, it is often difcult to obtain enough decision data to compute human performance in similar task instances, especially for a new task.”

  • The paper relies on this without support.

    The estimates of humans' and AI's capabilities are far from perfect, and relying on this information to assess AI's suggestions may have serious consequences, especially in high-risk areas.

    “we acknowledge that the estimates of humans' and AI's capabilities are far from perfect, and relying on this information to assess AI's suggestions may have serious consequences, especially in high-risk areas.”

  • The paper relies on this without support.

    When AI is wrong, lower agreement with AI is better.

    “Note that when AI is wrong, a lower agreement with AI is better.”

  • The authors state this as an assumption.

    In the absence of correctness feedback, a user's decision model remains relatively fixed in the short term.

    “We assume that in the absence of correctness feedback (e.g., no access to ground truth), the user's decision model is relatively fxed in the short term, which is reasonable for our experiments.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Pilot studies indicate that a few calibration sessions can greatly reduce how often reviewers accept model suggestions without question.

    “Empirical pilots show that even a handful of sessions sharply reduces blind acceptance of model suggestions.29”

    From When a Human‐in‐the‐Loop Amplifies the Risk of Misalignment: Considerations for Research Ethics Oversight (2026) · cites this paper as [29]

    Finding 3 reports that participants using the proposed interfaces agreed less often with incorrect AI recommendations than participants in the AI-confidence comparison condition. But the study tested interface designs during a single decision-making experiment, not whether a handful of calibration sessions reduced blind acceptance over time; its participants were not described as reviewers (Participants, §4.2.2; Finding 3, page unavailable). (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Cites

Flagged first, then by citations

Research works printed but not linked: 33

  1. Donald M Berwick, Harvey V Fineberg, and Milton C Weinstein. 1981. When doctors meet numbers. The American journal of medicine 71, 6 (1981), 991–998.
  2. John Brooke 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7.
  3. Leda Cosmides and John Tooby. 1996. Are humans good intuitive statisticians after all? Rethinking some conclusions from the literature on judgment under uncertainty. cognition 58, 1 (1996), 1–73.
  4. Marc Damez, Thanh Ha Dang, Christophe Marsala, and Bernadette Bouchon-Meunier. 2005. Fuzzy decision tree for user modeling from human-computer interactions. In Proceedings of the 5th International Conference on Human System Learning, ICHSL, Vol. 5. 287–302.
  5. Steven E Dilsizian and Eliot L Siegel. 2014. Artificial intelligence in medicine and cardiac imaging: harnessing big data and advanced computing to provide personalized medical diagnosis and treatment. Current cardiology reports 16, 1 (2014), 1–8.
  6. Jessica Maria Echterhoff, Matin Yarmand, and Julian McAuley. 2022. AI-Moderated Decision-Making: Capturing and Balancing Anchoring Bias in Sequential Decision Tasks. In CHI Conference on Human Factors in Computing Systems. 1–9.
  7. Nicholas Epley and Thomas Gilovich. 2001. Putting adjustment back in the anchoring and adjustment heuristic: Differential processing of self-generated and experimenter-provided anchors. Psychological science 12, 5 (2001), 391–396.
  8. Riccardo Fogliato, Shreya Chappidi, Matthew Lungren, Paul Fisher, Diane Wilson, Michael Fitzke, Mark Parkinson, Eric Horvitz, Kori Inkpen, and Besmira Nushi. 2022. Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 1362–1374.
  9. Adrian Furnham and Hua Chu Boo. 2011. A literature review of the anchoring effect. The journal of socio-economics 40, 1 (2011), 35–42.
  10. Krzysztof Z Gajos and Lena Mamykina. 2022. Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning. In 27th International Conference on Intelligent User Interfaces. 794–806.
  11. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning. PMLR, 1321–1330.
  12. Hsiu-Fang Hsieh and Sarah E Shannon. 2005. Three approaches to qualitative content analysis. Qualitative health research 15, 9 (2005), 1277–1288.
  13. Philip Nicholas Johnson-Laird. 1983. Mental models: Towards a cognitive science of language, inference, and consciousness. Number 6. Harvard University Press.
  14. Daniel Kahneman. 2011. Thinking, fast and slow. Macmillan.
  15. John D Lee and Neville Moray. 1994. Trust, self-confidence, and operators’ adaptation to automation. International journal of human-computer studies 40, 1 (1994), 153–184.
  16. John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
  17. Shuai Ma, Mingfei Sun, and Xiaojuan Ma. 2022. Modeling Adaptive Expression of Robot Learning Engagement and Exploring its Effects on Human Teachers. ACM Transactions on Computer-Human Interaction (2022).
  18. David Madras, Toni Pitassi, and Richard Zemel. 2018. Predict responsibly: improving fairness and accuracy by learning to defer. Advances in Neural Information Processing Systems 31 (2018).
  19. Ashley ND Meyer, Velma L Payne, Derek W Meeks, Radha Rao, and Hardeep Singh. 2013. Physicians’ diagnostic accuracy, confidence, and resource requests: a vignette study. JAMA internal medicine 173, 21 (2013), 1952–1958.
  20. Deborah J Miller, Elliot S Spengler, and Paul M Spengler. 2015. A meta-analysis of confidence and judgment accuracy in clinical decision making.Journal of Counseling Psychology 62, 4 (2015), 553.
  21. Don A Moore. 2020. Perfectly confident: How to calibrate your decisions wisely. HarperCollins.
  22. Andrew Y Ng Stuart Russell 2000. Algorithms for inverse reinforcement learning.. In Icml Vol. 1. 2.
  23. Raymond S Nickerson. 1998. Confirmation bias: A ubiquitous phenomenon in many guises. Review of general psychology 2, 2 (1998), 175–220.
  24. Mahsan Nourani, Chiradeep Roy, Jeremy E Block, Donald R Honeycutt, Tahrima Rahman, Eric Ragan, and Vibhav Gogate. 2021. Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems. In 26th International Conference on Intelligent User Interfaces. 340–350.
  25. Raja Parasuraman and Victor Riley. 1997. Humans and automation: Use, misuse, disuse, abuse. Human factors 39, 2 (1997), 230–253.
  26. Ellen Peters, Daniel Västfjäll, Paul Slovic, CK Mertz, Ketti Mazzocco, and Stephan Dickert. 2006. Numeracy and decision making. Psychological science 17, 5 (2006), 407–413.
  27. John Platt 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers 10, 3 (1999), 61–74.
  28. Amy Rechkemmer and Ming Yin. 2022. When Confidence Meets Accuracy: Exploring the Effects of Multiple Performance Indicators on Trust in Machine Learning Models. In CHI Conference on Human Factors in Computing Systems. 1–14.
  29. Valerie F Reyna and Charles J Brainerd. 2008. Numeracy, ratio bias, and denominator neglect in judgments of risk and probability. Learning and individual differences 18, 1 (2008), 89–107.
  30. Paul Slovic and Ellen Peters. 2006. Risk perception and affect. Current directions in psychological science 15, 6 (2006), 322–325.
  31. Yan-Yan Song and LU Ying. 2015. Decision tree methods: applications for classification and prediction. Shanghai archives of psychiatry 27, 2 (2015), 130.
  32. Peter C Wason and J St BT Evans. 1974. Dual processes in reasoning?Cognition 3, 2 (1974), 141–154.
  33. Nathan Weber and Neil Brewer. 2004. Confidence-accuracy calibration in absolute and relative face recognition judgments.Journal of Experimental Psychology: Applied 10, 3 (2004), 156.

Cited by

Sorted by citations

Related

Authors
Ma, S., Lei, Y., Wang, X., Zheng, C., Shi, C., Yin, M., & Ma, X.
Affiliations
  • Hong Kong University of Science and Technology (HK)
  • East China Normal University (CN)
  • Purdue University West Lafayette (US)
ORCID
Venue
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2023-04-19
Pages
1-19
DOI
10.1145/3544548.3581058
OpenAlex
W4366547597
Semantic Scholar
f61e4f6126e412a46dd1c9160eb97729fa435ace
Cited by
143
References
73
Citations by year
2026: 54 · 2025: 52 · 2024: 32 · 2023: 5
FWCI
71.46
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
General Research Fund (GRF) (16203421)
Topics
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
trust in AI, AI-assisted decision-making, AI confidence, trust calibration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review