CoChat
Sign Up
Cited by 172Explainable Artificial Intelligence (XAI)Ethics and Social Impacts of AIHuman-Automation Interaction and Safety

Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI

Open access
Miller, Tim·2023·2023 ACM Conference on Fairness Accountability and Transparency
DOI 10.1145/3593013.3594001
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we argue for a paradigm shift from the current model of explainable artificial intelligence (XAI), which may be counter-productive to better human decision making. In early decision support systems, we assumed that we could give people recommendations and that they would consider them, and then follow them when required. However, research found that people often ignore recommendations because they do not trust them; or perhaps even worse, people follow them blindly, even when the recommendations are wrong. Explainable artificial intelligence mitigates this by helping people to understand how and why models give certain recommendations. However, recent research shows that people do not always engage with explainability tools enough to help improve decision making. The assumption that people will engage with recommendations and explanations has proven to be unfounded. We argue this is because we have failed to account for two things. First, recommendations (and their explanations) take control from human decision makers, limiting their agency. Second, giving recommendations and explanations does not align with the cognitive processes employed by people making decisions. This position paper proposes a new conceptual framework called Evaluative AI for explainable decision support. This is a machine-in-the-loop paradigm in which decision support tools provide evidence for and against decisions made by people, rather than provide recommendations to accept or reject. We argue that this mitigates issues of over- and under-reliance on decision support tools, and better leverages human expertise in decision making.

AI snapshot

Written from the paper’s full text.

TL;DR

This position paper argues that recommendation-driven explainable AI ("recommend and defend") is counterproductive for decision support because it removes user agency and mismatches human cognitive processes, and proposes instead a conceptual framework called Evaluative AI, in which tools provide evidence for and against the decision maker's own hypotheses rather than recommendations. The authors frame this as a machine-in-the-loop, hypothesis-driven paradigm that retains explainable AI's tools while shifting control to the human.

Key findings

  • The paper's central claim is that current XAI decision support fails because (1) recommendations and their explanations take control from decision makers, limiting agency, and (2) they do not align with the cognitive processes people actually use when deciding.
  • An informal survey conducted by the author during a talk found "just three out of over 100 people preferred Bluster" (the recommendation-giving friend), with the remaining 100+ preferring Prudence (who gives evidence for and against the user's own proposals).
  • The authors report that experimental studies indicate most participants do not cognitively engage with explainability tools, and that those who do may become over-confident, even when explanations contain no useful information; they also cite Buçinca et al. and Gajos and Mamykina showing explainability did not mitigate over-reliance and in some cases increased it.
  • Evaluative AI is claimed to support options, judgement, understanding, and trade-offs (per Table 4), whereas recommendation, XAI, and interpretability paradigms provide only partial or no support for options and trade-offs; cognitive forcing is described as an improvement but still a "recommend and defend" approach.

Methods

  • Design: a position/conceptual paper presenting a new framework (Evaluative AI) rather than an empirical study; it reviews related work on cognitive decision making, abductive reasoning, the Data/Frame theory of sensemaking, and four dominant decision-support paradigms (recommendations; recommendations with explanations; interpretable models; cognitive forcing).
  • Data/sample: no primary dataset; the only original data reported is an informal audience survey (3 of 100+ preferring the recommendation-giving "Bluster"), plus a referenced but unpublished observational study of expert board gamers using an AI recommendation system.
  • Analysis: conceptual comparison of the five paradigms against six criteria for good decision support derived from Yates and Potworowski's 10 "cardinal decision issues," summarized in Tables 2 and 4, plus an illustrative skin-cancer diagnosis example (ISIC 2018, seven disease categories).

Limitations

  • Stated by the paper: if people dismiss recommendations and explainability information, they may equally fail to engage with evidence-based support; the framework makes the same assumption that people care what a machine says.
  • Stated by the paper: evaluative AI will likely increase cognitive load and force more engagement, which decision makers may prefer less (citing Buçinca et al.), and balancing information reduction against engagement is a challenge.
  • Evident from the text: the framework is conceptual and unevaluated — no empirical test of Evaluative AI is reported, the supporting survey is informal and non-peer-reviewed, and one cited observational study is described as "currently unpublished."

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3593013.3594001

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 83 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In an unpublished observational study with expert board gamers using an AI-based recommendation system, many players did not truly engage with recommendations or explanatory information.

    “In a study where expert board gamers used an AI-based recommendation system, we saw that many players did not truly engage with recommendations or explanatory information either.”
    Quote in the paperSubstantial claim
  • Interpretationreasoning

    'Recommend and defend' approaches do little to help us critique its answers or our own ideas.

    “'Recommend and defend' approaches, which we call recommendation-driven decision support, do little to help us critique its answers or our own ideas.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Current AI decision aids are designed to tell the user what the authors think the best answer is and explain why that is considered the best answer.
  • Premisecitation

    People often ignore AI recommendations because they do not trust them.

    “research found that people often ignore recommendations because they do not trust them”
    • Also stated in: § 3.1 Giving recommendations with no explanatory information · plain
      “people tend to ignore this due to unwarranted distrust [9,10,13,26,37,39]”
    Quote in the paperSubstantial claim0 of 6 sources checked
    • cites [9] source

      Not checked in the free run.

    • cites [10]

      Not checked in the free run.

    • cites [13] source

      Not checked in the free run.

    • cites [26]

      Not checked in the free run.

    • cites [37] source

      Not checked in the free run.

    • cites [39] source

      Not checked in the free run.

  • Interpretationcitation

    The Prudence-like approach is more effective for decision support because it aligns with the cognitive decision-making process that people use when making judgements and decisions.

    “We argue that the reason this Prudence-like approach is more effective for decision support because it aligns with the cognitive decisionmaking process that people use when making judgements and decisions [11,17,30].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [11]

      Not checked in the free run.

    • cites [17]

      Not checked in the free run.

    • cites [30]

      Not checked in the free run.

  • Premisecitation

    People sometimes follow AI recommendations blindly even when the recommendations are wrong.

    “or perhaps even worse, people follow them blindly, even when the recommendations are wrong.”
    • Also stated in: § 3.1 Giving recommendations with no explanatory information · plain
      “accept wrong decisions due to unwarranted trust [1,6,29]”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [1] source

      Not checked in the free run.

    • cites [6] source

      Not checked in the free run.

    • cites [29] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. 36
  • May combine two claims. 34

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In an informal survey conducted by the author, just three out of over 100 people preferred Bluster; the remaining 100+ preferred Prudence.

    “An informal survey conducted by the author during a recent talk showed that just three out of over 100 people preferred Bluster; the remaining 100+ people preferring Prudence.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In that study, when players did engage, there were some behavioural changes but not enough to see a statistically significant improvement in decision making.

    “When they did, we saw some behavioural changes, but not enough to see a statistically significant improvement in decision making.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    Evaluative AI is more suitable for medium- and high-stakes decisions when the decision maker is ultimately accountable, and low frequency decision making where the decision maker has time to explore options.

    “It is more suitable for medium-and high-stakes decisions when the decision maker is ultimately accountable, and low frequency decision making where the decision maker has time to explore options.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Evaluative AI mitigates fixation by either allowing the decision maker to determine which options are best or helping them to narrow down to a manageable set of options.

    “They mitigate fixation by either allowing the decision maker to determine which options are best or helping them to narrow down to a manageable set of options [34].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [34]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Both under- and overreliance have negative consequences.

    “Both under-and overreliance have negative consequences [29].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [29] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The effect of system 1 thinking may differ between contexts such as the experience of the decision maker and the stakes of the decision.

    “The effect of system 1 thinking may differ between contexts such as the experience of the decision maker and the stakes of the decision.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Evaluative AI could prove useful by not fixating people on particular recommendations, but allowing people to assess whether evidence supports their hypotheses, rather than trying to understand the DA's reasoning.

    “This indicates that Evaluative AI could prove useful by not fixating people on particular recommendations, but allowing people to assess whether evidence supports their hypotheses, rather than trying to understand the DA's reasoning.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Giving only recommendations does not help to provide new options or to filter out unlikely options.

    “Giving only recommendations: (1) does not help to provide new options or to filter out unlikely options”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Participants who do engage with explainability tools seem to believe they understand explanatory information, leading to over-confidence, even when explanations contain no useful information.

    “Participants who do seem to believe that they understand explanatory information, leading to over-confidence [2], even when presented with explanations that contain no useful information [5].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [2] source

      Not checked in the free run.

    • cites [5] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Despite cognitive forcing functions being more effective, they were least preferred by participants.

    “Interestingly, despite cognitive forcing functions being more effective, they were least preferred by participants.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [1] source

      Not checked in the free run.

    • cites [6] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Buçinca et al. and Gajos and Mamykina show that explainability did not mitigate over-reliance, and in some cases increased over-reliance.

    “Similarly, in experiments on a general task with general participants, Buçinca et al. [1] and Gajos and Mamykina [6] show that explainability did not mitigate over-reliance, and in some cases it increased over-reliance.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [1] source

      Not checked in the free run.

    • cites [6] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Evaluative AI is still a form of explainable AI.

    “but that the evaluative AI paradigm is still a form of explainable AI”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Many elements of the current XAI paradigm will play a part in the evaluative AI conceptualisation.

    “and many of the current paradigm will play a part in this new conceptualisation; so, long live explainable AI!”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Premisecitation

    Understandability is important to calibrate trust and reliance, and to find mistakes.

    “This is a common criteria for AI-based DAs [9], and is important to calibrate trust & reliance, and to find mistakes.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Recent research has shown that recommendation-driven explainable/interpretable AI has little effect on decision making, although this is not always the case.

    “However, recent research has shown that the recommendation-driven explainable/interpretable AI has little effect on decision making [1, 6-8, 28, 32], although this is not always the case [22,23,41].”
    Quote in the paperSubstantial claim0 of 9 sources checked
    • cites [1] source

      Not checked in the free run.

    • cites [6] source

      Not checked in the free run.

    • cites [7] source

      Not checked in the free run.

    • cites [8] source

      Not checked in the free run.

    • cites [28] source

      Not checked in the free run.

    • cites [32] source

      Not checked in the free run.

    • cites [22]

      Not checked in the free run.

    • cites [23] source

      Not checked in the free run.

    • cites [41] source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Abductive reasoning plays a central role in the Data/Frame decision process.

    “Klein et al. [17] argue that abductive reasoning plays a central role in this process.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [17]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Overreliance means decision makers accept machine recommendations even when wrong, caused by unwarranted trust.

    “Overreliance means decision makers accept a machine recommendations even when it is wrong, caused by unwarranted trust [13].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Cognitive forcing slightly mitigated overreliance compared to feature-based and uncertainty judgements, particularly disregarding incorrect AI recommendations, but with no statistically significant differences in task performance.

    “Their results showed that cognitive forcing slightly mitigated overreliance, compared to feature-based and uncertainty judgements, particularly disregarding incorrect AI recommendations, but with no statistically significant differences in task performance.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [1] source

      Not checked in the free run.

    • cites [6] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Explainability provides understanding.

    “So, explainability provides understanding.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Under-reliance means decision makers reject a machine recommendation even when correct, caused by unwarranted distrust.

    “Under-reliance means decision makers reject a machine recommendation even when it is correct, caused by unwarranted distrust [13].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Abductive reasoning is the process of forming hypotheses and judging their likelihood for the purpose of explaining observations or facts.

    “Abductive reasoning is the process of forming hypotheses and judging their likelihood for the purpose of explaining observations or facts [30].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Decision makers need to 'invert' the explainable AI process to align with the decision-making process.

    “Decision makers need to 'invert' the explainable AI process to align with the decisionmaking process.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Giving only recommendations does not consider possibilities or judgements beyond one option.

    “(2) does not consider possibilities or judgements beyond one option”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Decision making is a process, not a point in time.

    “However, as a model, it misses some important aspects of decision support; importantly: that decision making is a process, not a point in time.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Cognitive forcing does not provide helpful decision-making information for options other than the recommendation.

    “Second, it does not provide helpful decision-making information for options other than the recommendation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Evaluative AI provides support for options, judgement, understanding, and trade-offs.

    “Evaluative AI explicitly aims to provide support for options, judgement, understanding, and trade-offs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    Evaluative AI explicitly hands control of which hypotheses are investigated and prioritised to the decision maker, resulting in a machine-in-the-loop paradigm rather than a human-in-the-loop paradigm.

    “(2) Control: Evaluative AI explicitly hands control of which hypotheses are investigated and prioritised to the decision maker, resulting a machine-in-the-loop paradigm [7,8], rather than a human-in-the-loop paradigm.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [7] source

      Not checked in the free run.

    • cites [8] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Giving explanatory information from the start may help the decision maker to focus on useful information.

    “Giving explanatory information from the start may help the decision maker to focus on useful information.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    New models of XAI are needed, but existing work lays a solid foundation.

    “New models of XAI are needed, but existing work lays a solid foundation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    For a machine to judge decisions, an underlying decision-making model is needed, so recommendation approaches such as machine learning, planning, and optimisation will be required, along with XAI techniques.

    “Second, for a machine to judge decisions, we will need an underlying decision-making model, meaning that recommendation approaches such as machine learning, planning, and optimisation will be required, along with XAI techniques.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    The current paradigm of explainable AI as justified recommendations is 'dead', but a new paradigm that includes hypothesis-driven explainability could take its place.

    “We conclude by repeating that the current paradigm of explainable AI as justified recommendations is 'dead'.But the new paradigm that includes hypothesis-driven explainability could take the throne, so long live explainable AI!”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: People care at all about what a machine has to say.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    Evaluative AI provides decision makers with better control and follows a process they will naturally follow, unlike recommendation-driven approaches which disrupt the decision-making process.

    “However, one argument against this is that this evaluative AI provides decision makers with better control [36,43], and follows a process that they will naturally follow, rather than recommendation-driven approaches, which somewhat disrupt the decision-making process.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [36]

      Not checked in the free run.

    • cites [43] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Evaluative AI is built on the Data/Frame model of sensemaking, so it supports the decision maker's cognitive process by allowing them to explore hypotheses, rather than providing only information to justify a machine recommendation.

    “(1) Cognitive processes: Evaluative AI is built on the Data/Frame model of sensemaking [17], so supports the decision maker's cognitive process by allowing them to explore hypotheses,”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [17]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisenone given

    There are applications where recommendation-driven approaches may be the best way to improve decisions, such as making decisions at scale.

    “Further, there are applications where recommendation-driven approaches may be the best way to improve decisions, such as making decisions at scale.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    Evaluative AI is explainable AI.

    “Third, evaluative AI is explainable AI.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Evaluative AI obtains the benefits of cognitive forcing while giving the decision maker control to explore the strengths and weaknesses of any option, rather than only the strengths of recommendations and weaknesses of alternatives.

    “For this reason, we assert that evaluative AI obtains the benefits of cognitive forcing, while giving control to the decision maker to explore the strengths and weaknesses of any option, rather than just the strengths recommendations and the weaknesses of alternatives.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: People care at all about what a machine has to say.

  • May combine two claims.

    Interpretationreasoning

    Cognitive forcing partially helps to provide new options or to filter out unlikely options by forcing the decision maker to do so.

    “(1) partially helps to provide new options or to filter out unlikely options by forcing the decision maker to do so”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Cognitive forcing does provide understanding of the machine decision.

    “and (5) does provide understanding of the machine decision.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    Bluster can be useful in low stakes decisions or when time is limited, but overall Bluster does not support our cognitive process as well as Prudence.

    “Bluster can be useful; for example, in low stakes decisions or if time is limited; but overall, Bluster does not support our cognitive process as well as Prudence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Drury et al. proposed a framework in which machines consider their own option awareness to support human decision makers.

    “More recently, Drury et al. [4] proposed a framework in which machines consider their own option awareness to support human decision makers.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Prudence makes us do more work, which people prefer to avoid, so participants may prefer less work but with worse results.

    “Our friend Prudence makes us do more work, which people prefer to avoid [19], so we may find that participants prefer less work, but with worse results Buçinca et al. [1].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [19] source

      Not checked in the free run.

    • cites [1] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Withholding recommendations forces the decision maker to cognitively engage with the decision and therefore to consider different options and make trade-offs.

    “Withholding recommendations 'forces' the decision maker to cognitively engage with the decision and therefore, to consider different options and make tradeoffs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    Bluster and Prudence can both be useful in supporting decisions, but Prudence puts us in control and is better at helping us weigh up different options.

    “Our friends Bluster and Prudence can both be useful in supporting our decisions, but Prudence puts us in control and is better at helping us weigh up different options.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: People care at all about what a machine has to say.

  • May combine two claims.

    Interpretationcitation

    The cognitive and social aspects of good explainability apply to evaluative AI: explanations are contrastive, selected, interactive, and causal.

    “The cognitive and social aspects of good explainability apply to evaluative AI: [26]: (1) explanations are contrastive; (2) explanations are selected; (3) explanations are interactive; and (4) explanations are causal.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [26]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    There are applications of XAI beyond decision support, such as verification, regulation, scientific discovery, and generating insights about underlying data.

    “First, there are applications of XAI beyond decision support, such as verification, regulation, scientific discovery, generating insights about underlying data, etc.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Pfaff et al. showed that using visual analytics to let decision makers explore options increases their option awareness, leading to more accurate and faster decisions.

    “Pfaff et al. [31] show that using visual analytics to allow decision makers to explore options increases their option awareness, resulting in more accurate and faster decisions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Cognitive forcing does not consider possibilities or judgements beyond one option.

    “(2) does not consider possibilities or judgements beyond one option”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Evaluative AI supports both evidence for and against options, rather than aiming to persuade that the machine is correct.

    “Evaluative AI supports both, rather than aiming to persuade that the machine is correct.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    In evaluative AI, the decision maker keeps control over which hypotheses to explore.

    “Importantly, the decision maker to maintains control over which hypotheses to explore.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Evaluative AI aligns with decision making processes, keeps the decision maker in control, and asks users to rely on evidence instead of recommendations.

    “A model of Evaluative AI, which aligns with decision making processes, keeping the decision maker in control and asking users to rely on evidence instead of recommendations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Evaluative AI will likely result in designs that force more engagement with the decision, but are less preferred by decision makers.

    “Evaluative AI will likely result in designs that force more engagement with the decision, but less preferred by decision makers [1].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Rudin argues that for high-stakes decisions, people should avoid using black-box models altogether, and instead should use interpretable models.

    “Rudin [33] argues that for high-stakes decisions, people should avoid using black-box models altogether, and instead should use interpretable models.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    Evaluative AI does not necessarily provide recommendations; instead it supports filtering out unlikely options, generating new hypotheses, or both.

    “Evaluative AI does not (necessarily) provide recommendations, but instead offers support to filter out unlikely options, generate new hypotheses, or both.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Interpretable models use a small set of features with a low complexity of interaction between them.

    “Interpretable models use a small set of features with a low complexity of interaction between them.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [33] source

      Not checked in the free run.

54 of 54

  • The authors state this as an assumption.

    The initial assumption of AI-assisted decision aids was that people would follow recommendations when required, leading to better decisions.

    “The initial assumption of AI-assisted DAs was that people would follow recommendations when required, leading to better decisions.”

  • The authors state this as an assumption.

    For situations in which decision aids can handle many more factors simultaneously, they can provide insight when the assumptions of the underlying modelling theory hold.

    “The assumption here is that for situations in which DAs can handle many more factors simultaneously, it can provide insight when the assumptions of the underlying modelling theory hold.”

  • The authors state this as an assumption.

    Decision makers will carefully consider recommendations.

    “This assumes that decision makers will carefully consider recommendations.”

  • The authors state this as an assumption.

    Because an interpretable model is interpretable, it contains most of the explanatory information that is required.

    “The assumption is that because the model is interpretable, it contains most of the explanatory information that is required.”

  • The authors state this as an assumption.

    Forcing people to engage with the decision cognitively can mitigate over-reliance.

    “The assumption of cognitive forcing is that forcing people to engage with the decision cognitively can mitigate over-reliance.”

  • The paper relies on this without support.

    Current AI decision aids are designed to tell the user what the authors think the best answer is and explain why that is considered the best answer.

    “AI decision aids are designed to tell the user what they think the best answer is (e.g. a recommendation), and explain why that is considered the best answer.”

  • The authors state this as an assumption.

    People care at all about what a machine has to say.

    “The evaluative AI framework makes the same assumptions as earlier work: that people care at all about what a machine has to say.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Cites

Flagged first, then by citations

Research works printed but not linked: 10

  1. Robert R Hoffman. 2017. A taxonomy of emergent trusting in the human–machine relationship. Cognitive systems engineering: The future for a changing world (2017), 137–164.
  2. Robert R Hoffman, Tim Miller, and William J Clancey. 2022. Psychology and AI at a Crossroads: How Might Complex Systems Explain Themselves?The American journal of psychology 135, 4 (2022), 365–378.
  3. Daniel Kahneman. 2011. Thinking Fast and Slow. Penguin UK.
  4. Harmanpreet Kaur, Eytan Adar, Eric Gilbert, and Cliff Lampe. 2022. Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory. (May 2022). arxiv:2205.05057 [cs.HC] http://arxiv.org/abs/2205.05057 · arXiv
  5. Gary Klein, Jennifer K Phillips, Erica L Rall, and Deborah A Peluso. 2007. A data–frame theory of sensemaking. In Expertise out of context. Psychology Press, 118–160.
  6. 10.31219/osf.io · DOI
  7. David Alvarez Melis, Harmanpreet Kaur, Hal Daumé, III, Hanna Wallach, and Jennifer Wortman Vaughan. 2021. From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 35–47. https://ojs.aaai.org/index.php/HCOMP/article/view/18938
  8. Hugo Mercier and Dan Sperber. 2017. The Enigma of Reason. Harvard University Press.
  9. Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence (2019). https://www.sciencedirect.com/science/article/pii/S0004370218305988
  10. J W Rudolph. 2003. Into the big muddy and out again: Error persistence and crisis management in the operating room. Ph. D. Dissertation.

Cited by

Sorted by citations

Related

Authors
Miller, T.
Affiliations
The University of Melbourne (AU)
Corresponding author
Miller, Tim
ORCID
Miller, Tim
Venue
2023 ACM Conference on Fairness Accountability and Transparency
Publisher
ACM
Published
2023-06-12
Pages
333-342
DOI
10.1145/3593013.3594001
OpenAlex
W4380366109
Cited by
172
References
38
Citations by year
2026: 55 · 2025: 64 · 2024: 47 · 2023: 6
FWCI
30.53
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
  • Minerva Access (University of Melbourne) (submittedVersion)
Funders
Australian Research Council (DP190103414)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
Keywords
explainable AI, decision support system, human decision-making, recommendation trust, human agency, cognitive processes in decision making, overreliance on AI, human expertise

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review