Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care
Open accessThe DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index; none of the 31 sources its claims cite is checked for support yet.
Abstract
Artificial intelligence (AI) in healthcare has the potential to improve patient outcomes, but clinician acceptance remains a critical barrier. We developed a novel decision support interface that provides interpretable treatment recommendations for sepsis, a life-threatening condition in which decisional uncertainty is common, treatment practices vary widely, and poor outcomes can occur even with optimal decisions. This system formed the basis of a mixed-methods study in which 24 intensive care clinicians made AI-assisted decisions on real patient cases. We found that explanations generally increased confidence in the AI, but concordance with specific recommendations varied beyond the binary acceptance or rejection described in prior work. Although clinicians sometimes ignored or trusted the AI, they also often prioritized aspects of the recommendations to follow, reject, or delay in a process we term “negotiation.” These results reveal novel barriers to adoption of treatment-focused AI tools and suggest ways to better support differing clinician perspectives.
Overview
CoChat Summary
TL;DR
In a mixed-methods think-aloud study with 24 ICU clinicians using an AI Clinician-based sepsis treatment CDS, explanations increased perceived usefulness and confidence in the AI but did not change overall binary concordance with recommendations. Clinicians often engaged in partial reliance—“negotiation”—rather than simply ignoring or trusting the AI, revealing barriers for treatment-focused AI adoption.
Key findings
- Explanations improved perceptions: Feature Explanation was rated more useful than Text Only (Δ = 0.83, 95% CI [0.24, 1.43], p = 0.018) and had a more positive effect on confidence (Δ = 1.08, 95% CI [0.51, 1.66], p < 0.001); Alternative Treatments showed directionally similar benefits.
- Concordance with the AI did not vary meaningfully by visualization condition: clinicians chose the same treatment as the AI about 42% of the time across AI conditions, only slightly above the 33% base rate without seeing the recommendation. Full concordance with the majority attending decision averaged 50% across AI conditions versus 63% in the No AI condition.
- Four behavior patterns emerged: Ignore (7 participants, 21 decisions), Negotiate (12 participants, 36 decisions), Consider (3 participants, 9 decisions), and Trust/Rely (2 participants). Negotiate participants rated recommendations more useful (M = 4.6, SD = 1.27) than other groups (M = 2.8, SD = 1.79).
- Explanations increased perceived case difficulty: No AI was rated less challenging than Alternative Treatments (Δ = 1.08, 95% CI [0.46, 1.71], p = 0.003) and directionally less challenging than Feature Explanation (Δ = 0.79, 95% CI [0.14, 1.45], p = 0.09).
Methods
- Mixed-methods think-aloud study with 24 practicing ICU clinicians (attending physicians, advanced practice providers, and critical care fellows) from a large multihospital academic health system. Each participant made AI-assisted treatment decisions for four real de-identified MIMIC-IV sepsis cases, with four visualization conditions presented in fixed order: No AI, Text Only, Feature Explanation, and Alternative Treatments.
- The CDS was built on a replicated AI Clinician reinforcement learning model trained on 18,143 MIMIC-IV sepsis patients, using 750 patient states and 25 treatment actions (5 IV fluid levels and 5 vasopressor doses binned by quantiles). The displayed model had a policy value of 83.8, compared with original reported values of 80–90. Cases were selected where the AI recommendation substantially differed from historical clinician actions.
- Quantitative analysis used OLS regression for 7-point Likert outcomes and logistic regression for concordance, with clustered standard errors and Holm-Bonferroni post hoc adjustment. Qualitative analysis coded 12.6 hours of think-aloud transcripts using 23 codes, plus open coding and affinity diagramming.
Limitations
- Stated: case selection may have skewed observations toward particular cases and recommendations; patient depictions were limited to structured MIMIC-IV data; domain-specific explainable RL techniques were not used; SHAP explanations focused only on state clustering, limiting their potential usefulness.
- Stated: the study design prioritized rich think-aloud data, so statistical significance of some quantitative results, especially concordance, could not be assessed; the academic hospital setting may limit generalizability.
- Evident from the text: visualization conditions were presented in a fixed order, which may introduce order or learning effects; and because there is no objective ground truth for treatment decisions, concordance with the AI is not a direct measure of decision accuracy.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 17 Not checked · 80 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“two participants were infuenced by the AI for at least part of their decision in every decision they made”
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > 5.2.4
plainstates a causen = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants perceived the cases as being signifcantly less challenging in the No AI condition than in the Alternative Treatments condition”
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > Efect of AI on confdence.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants' confdence in their treatment choices was not signifcantly diferent across Visualization conditions”
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > Efect of AI on confdence.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Visualization condition afected how participants rated the AI's impact on their confdence”
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > Efect of AI on confdence.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In 12/36 decisions, participants used their perception of the severity of the patient's sepsis to decide how much to reconsider their treatment plan”
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > 5.2.2 Negotiate:
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
44 more claims
Create a free account to see all 97 claims and every source verdict.
- It may state two separate claims as one. 17
- The quote only partly states this claim. 4
- Probably not a substantial claim. 30
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Trust was driven more by evidence of external validation than interface features or explanation types.”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [62]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“research found that people often ignore recommendations because they do not trust them”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [37]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This study directly investigates clinician interaction, acceptance, and trust-including 'negotiation' behaviors-with an interpretable AI-CDSS for sepsis treatment [62].”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [62]
The cited source says what the claim attributes to it.
In the citing paper
“However, people find it difficult to correctly calibrate their trust in a decision aid [10,13,21,37].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [37]
The cited source says what the claim attributes to it.
In the citing paper
“Second, instead of full consensus or complete divergence, human and AI decision rationales often exhibit partial alignment [113, 128].”
From Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making (2025) · cites this paper as [113]
26 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > 6.2 Validating that AI-Based Decision Support Improves Outcomes
“the recent guidance by the U.S. Food and Drug Administration that treatment decision support systems such as the AI Clinician should be regulated as medical devices [81].”
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > References
“82. U.S. Food and Drug Administration. 2022. Clinical Decision Support Software. Technical Report. 1-26 pages. https://www.fda.gov/media/109618/download”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > 6 DISCUSSION
“for these participants the CDS only served to lower their confdence in their decision-making and increase the perceived difculty of the case.”
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > 5.2.1 Ignore: Participant makes own decisions.
“In these cases the AI served to either confrm the initial assessment-"made me feel better about that decision" (P5)-or, more commonly, to induce doubt when the recommendation was discordant (P6, P7, P17, P18).”
Citations
Cites
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569 · cited 1×
“For each of the 750 possible states, we trained an XGBoost classifer [19] to predict whether a patient was in that state or not.”
MethodA Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 5×
“In addition to the textual recommendation, participants were shown a SHAP feature attribution chart explaining how the patient’s state was determined (Fig.”
MethodInfluentialHigh-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · cited 1×
“The AI Clinician publication is widely known and highly infuential in both the critical care and CDS communities [79, 88].”
BackgroundSurviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock 2021.
Evans, L., Rhodes, A., Alhazzani, W., Antonelli, M., Coopersmith, C. M., French, C., Machado, F. R., Mcintyre, L., Ostermann, M., Prescott, H. C., et al. (2021) · Critical Care Medicine · doi:10.1097/ccm.0000000000005337 · Cited by 2,935
Shankar-Hari, M., Phillips, G. S., Levy, M. L., Seymour, C. W., Liu, V. X., Deutschman, C. S., Angus, D. C., Rubenfeld, G. D., Singer, M., & for the Sepsis Definitions Task Force (2016) · JAMA · doi:10.1001/jama.2016.0289 · Cited by 2,424
Cecconi, M., Evans, L., Levy, M., & Rhodes, A. (2018) · The Lancet · doi:10.1016/s0140-6736(18)30696-2 · Cited by 2,254
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · cited 3×
“However, none of these methods appear to work universally across contexts, particularly in health care; in some domains, they may infate or undermine confdence [31, 40, 90], while in others they may be poorly-aligned with human decision-making processes [2, 32].”
BackgroundDeep learning-enabled medical computer vision
Esteva, A., Chou, K., Yeung, S., Naik, N., Madani, A., Mottaghi, A., Liu, Y., Topol, E., Dean, J., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00376-2 · Cited by 1,406 · cited 1×
“Taking an alternative strategy, some AI systems instead strive to provide clinicians with useful non-prescriptive information, such as highlighting informative parts of a medical image [27], displaying information from similar historical cases [13], or identifying patients at high risk of future deterioriation [35, 68…”
BackgroundKomorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., & Faisal, A. A. (2018) · Nature Medicine · doi:10.1038/s41591-018-0213-5 · Cited by 1,290 · cited 4×
“[50] but that has not been prospectively evaluated in a clinical setting.”
BackgroundMethodInfluentialBuçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · Nature Medicine · doi:10.1038/s41591-022-01772-9 · Cited by 1,039
Human-computer collaboration for skin cancer recognition
Tschandl, P., Rinner, C., Apalla, Z., Argenziano, G., Codella, N., Halpern, A., Janda, M., Lallas, A., Longo, C., Malvehy, J., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-0942-0 · Cited by 1,024 · cited 5×
“either agreement or disagreement with a diagnosis) [14, 16, 35, 45, 80, 83].”
BackgroundMethodResultInfluentialDesigning Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906 · cited 1×
“Explanations can also interact with reasoning fallacies such as confirmation and availability bias [21, 45], but mitigating these effects requires knowledge of a normatively correct reasoning process [84] that may not always be available.”
BackgroundZhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895 · cited 7×
“The design of explainable and interpretable ML-based tools has become a major focus of research in the HCI community [26, 76, 90].”
BackgroundMethodResultInfluentialMarkus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754 · cited 1×
“In concert with human-centered design methods, these technical approaches can be integrated into algorithmic systems with the intent of improving trust and human-AI team performance [5, 37, 67].”
BackgroundTwo Decades of Mortality Trends Among Patients With Severe Sepsis: A Comparative Meta-Analysis*
Stevenson, E. K., Rubenstein, A. R., Radin, G. T., Wiener, R. S., & Walkey, A. J. (2014) · Critical Care Medicine · doi:10.1097/ccm.0000000000000026 · Cited by 648 · cited 1×
“As such, recommendations face signifcant challenges in translation to wider clinical practice [75], resulting in substantial variability in care practices [6] and continued high mortality levels [74].”
BackgroundReinforcement Learning in Healthcare: A Survey
Yu, C., Liu, J., Nemati, S., & Yin, G. (2023) · ACM Computing Surveys · doi:10.1145/3477600 · Cited by 628 · cited 1×
“The AI Clinician publication is widely known and highly influential in both the critical care and CDS communities [79, 88].”
BackgroundBeede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376718 · Cited by 614 · cited 1×
“For example, AI-driven CDS systems for image-based diagnosis have been increasingly accepted as tools to reduce clinician burden and prioritize attention [9, 80].”
MethodCai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 4×
“Unlike much prior work on AI-assisted decision-making in health care [14, 40, 78, 87], this study (1) used a real model trained to op-”
BackgroundMethodInfluentialVariations in Physician Practice: The Role of Uncertainty
Eddy, D. M. (1984) · Health Affairs · doi:10.1377/hlthaff.3.2.74 · Cited by 569 · cited 1×
“For a large number of health care decisions there is no evident “right answer” [23].”
BackgroundDo as AI say: susceptibility in deployment of clinical decision-aids
Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S. J., Lermer, E., Coughlin, J. F., Guttag, J. V., Colak, E., & Ghassemi, M. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00385-9 · Cited by 543 · cited 4×
“that acceptance of AI recommendations is often more strongly determined by the clinician’s expertise than by the quality of the recommendation [31, 80].”
BackgroundResultInfluentialThe “inconvenient truth” about AI in healthcare
Panch, T., Mattie, H., & Celi, L. A. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0155-4 · Cited by 473
The myth of generalisability in clinical research and machine learning in health care
Futoma, J., Simons, M., Panch, T., Doshi-Velez, F., & Celi, L. A. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30186-2 · Cited by 467 · cited 1×
“AI-based tools that seek to improve decision-making across diverse deployment contexts must produce recommendations that are both acceptable to health care providers and transparent in the case of errors [30, 82].”
BackgroundThe Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441 · cited 4×
“In terms of reliance on the AI, prior work studying reliance on CDS tools [12] and explainable AI [85, 90] led us to expect that clinicians would calibrate their own certainty against the AI and make a binary decision about whether”
BackgroundResultInfluentialJussupow, E., Spohrer, K., Heinzl, A., & Gawlitza, J. (2021) · Information Systems Research · doi:10.1287/isre.2020.0980 · Cited by 422 · cited 2×
“Explanations can also interact with reasoning fallacies such as confrmation and availability bias [21, 45], but mitigating these efects requires knowledge of a normatively correct reasoning process [84] that may not always be available.”
BackgroundHuman-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405 · cited 1×
“Taking an alternative strategy, some AI systems instead strive to provide clinicians with useful non-prescriptive information, such as highlighting informative parts of a medical image [27], displaying information from similar historical cases [13], or identifying patients at high risk of future deterioriation [35, 68…”
BackgroundToo much, too little, or just right? Ways explanations impact end users' mental models
Kulesza, T., Stumpf, S., Burnett, M., Yang, S., Kwan, I., & Wong, W.-K. (2013) · 2013 IEEE Symposium on Visual Languages and Human Centric Computing · doi:10.1109/vlhcc.2013.6645235 · Cited by 388 · cited 1×
“’s fndings that complete explanations tend to help despite requiring increased cognitive efort [51].”
BackgroundReasons For Physicians Not Adopting Clinical Decision Support Systems: Critical Analysis
Khairat, S., Marc, D., Crosby, W., & Al Sanousi, A. (2018) · JMIR Medical Informatics · doi:10.2196/medinform.8912 · Cited by 335 · cited 1×
“Even with similar experience levels, clinicians express varying degrees of awareness (and skepticism) of how recommendations are generated [48, 87], but the efects of diferences in these attitudes have yet to be examined.”
BackgroundWang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320 · cited 7×
“A mixed-methods analysis of the think-aloud transcripts and structured decision responses showed that explanations improved clinicians’ perceptions of the AI’s usefulness and made them more confdent in their own decisions, a fnding that is consistent with prior literature [1, 80, 85].”
BackgroundMethodResultInfluentialWhat Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use
Tonekaboni, S., Joshi, S., McCradden, M. D., & Goldenberg, A. (2019) · arXiv · doi:10.48550/arxiv.1905.05134 · Cited by 318 · cited 2×
“Unlike much prior work on AI-assisted decision-making in health care [14, 40, 78, 87], this study (1) used a real model trained to optimize treatment decisions, (2) provided clinical experts with real de-identifed patient data, and (3) utilized a think-aloud protocol to capture further nuance beyond a multiple-choice…”
BackgroundMethodSendak, M. P., Ratliff, W., Sarro, D., Alderton, E., Futoma, J., Gao, M., Nichols, M., Revoir, M., Yashar, F., Miller, C., et al. (2020) · JMIR Medical Informatics · doi:10.2196/15182 · Cited by 287 · cited 2×
“to sepsis, early-warning systems such as Sepsis Watch and the Targeted Real-time Early Warning System (TREWS) have been accepted by clinicians at the hospitals where they are deployed [35, 69], despite being initially met with ambivalence [34].”
BackgroundUpdates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff
Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012429 · Cited by 284 · cited 1×
“Another source of complication in validating AI-based recommendation systems is that reliability may vary signifcantly across diferent patient subgroups, requiring the user to develop a mental model of the AI’s error boundaries [4].”
BackgroundSepsis Among Medicare Beneficiaries: 1. The Burdens of Sepsis, 2012–2018*
Buchman, T. G., Simpson, S. Q., Sciarretta, K. L., Finne, K. P., Sowers, N., Collier, M., Chavan, S., Oke, I., Pennini, M. E., Santhosh, A., et al. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004224 · Cited by 265 · cited 1×
“hospitals [10], and as such it represents a major target for quality improvement eforts at the local and national level [36].”
BackgroundEddy, D. M. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03440030128037 · Cited by 261
Explainable Reinforcement Learning: A Survey
Puiutta, E., & Veith, E. M. S. P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-57321-8_5 · Cited by 252
Amann, J., Vetter, D., Blomberg, S. N., Christensen, H. C., Coffee, M., Gerke, S., Gilbert, T. K., Hagendorff, T., Holm, S., Livne, M., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000016 · Cited by 251 · cited 3×
“However, none of these methods appear to work universally across contexts, particularly in health care; in some domains, they may infate or undermine confdence [31, 40, 90], while in others they may be poorly-aligned with human decision-making processes [2, 32].”
BackgroundJacobs, M., Pradier, M. F., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01224-x · Cited by 247 · cited 7×
“used a mock AI system to evaluate clinician decisions on antidepressant selection [40], while Yang et al.”
BackgroundMethodResultInfluentialYang, Q., Steinfeld, A., & Zimmerman, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300468 · Cited by 247 · cited 4×
“Even with similar experience levels, clinicians express varying degrees of awareness (and skepticism) of how recommendations are generated [48, 87], but the efects of diferences in these attitudes have yet to be examined.”
BackgroundMethodInfluentialEarly Prediction of Sepsis in the ICU Using Machine Learning: A Systematic Review
Moor, M., Rieck, B., Horn, M., Jutzeler, C. R., & Borgwardt, K. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2021.607952 · Cited by 231 · cited 1×
“Multiple machine learning algorithms exist for mining hospital electronic health record data to identify patients with sepsis [58, 64, 77].”
BackgroundHuman factors in model interpretability: Industry practices, challenges, and needs
Hong, S. R., Hullman, J., & Bertini, E. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392878 · Cited by 219 · cited 1×
“concert with human-centered design methods, these technical approaches can be integrated into algorithmic systems with the intent of improving trust and human-AI team performance [5, 37, 67].”
BackgroundHuman-centered XAI: Developing design patterns for explanations of clinical decision support systems
Schoonderwoerd, T. A. J., Jorritsma, W., Neerincx, M. A., & van den Bosch, K. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102684 · Cited by 210 · cited 3×
“concert with human-centered design methods, these technical approaches can be integrated into algorithmic systems with the intent of improving trust and human-AI team performance [5, 37, 67].”
BackgroundWang, D., Wang, L., Zhang, Z., Wang, D., Zhu, H., Gao, Y., Fan, X., & Tian, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445432 · Cited by 209 · cited 2×
“[83] have identified clinician frustrations with the added workload of using a CDS, particularly when those systems do not adequately complement their expertise.”
BackgroundHenry, K. E., Kornfield, R., Sridharan, A., Linton, R. C., Groh, C., Wang, T., Wu, A., Mutlu, B., & Saria, S. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00597-7 · Cited by 207 · cited 6×
“[35, 69], despite being initially met with ambivalence [34].”
BackgroundInfluential"The human body is a black box": supporting clinical decision-making with deep learning
Sendak, M., Elish, M. C., Gao, M., Futoma, J., Ratliff, W., Nichols, M., Bedoya, A., Balu, S., & O'Brien, C. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372827 · Cited by 196 · cited 1×
“Taking an alternative strategy, some AI systems instead strive to provide clinicians with useful non-prescriptive information, such as highlighting informative parts of a medical image [27], displaying information from similar historical cases [13], or identifying patients at high risk of future deterioriation [35, 68…”
BackgroundGinestra, J. C., Giannini, H. M., Schweickert, W. D., Meadows, L., Lynch, M. J., Pavan, K., Chivers, C. J., Draugelis, M., Donnelly, P. J., Fuchs, B. D., et al. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003803 · Cited by 159 · cited 1×
“[35, 69], despite being initially met with ambivalence [34].”
BackgroundI Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI
Chromik, M., Eiband, M., Buchner, F., Krüger, A., & Butz, A. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450644 · Cited by 144 · cited 1×
“[12] to more recent work [20, 85, 90] has shown that these explanations tend to increase trust in the model even when it is unwarranted.”
BackgroundVasopressors in septic shock: which, when, and how much?
Shi, R., Hamzaoui, O., Vita, N. D., Monnet, X., & Teboul, J. (2020) · Annals of Translational Medicine · doi:10.21037/atm.2020.04.24 · Cited by 143 · cited 1×
“It is impossible to know if the AI’s recommendation to give vasopressors was a better decision, although some evidence shows that early administration of vasopressors could beneft patients [72] and expert opinion increasingly emphasizes vasopressors over fuid administration [44].”
BackgroundKawakami, A., Sivaraman, V., Cheng, H.-F., Stapleton, L., Cheng, Y., Qing, D., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517439 · Cited by 143 · cited 1×
“While expert consensus can serve as a useful proxy for the ground truth [40], the accuracy of a real-world decision is often fundamentally unknowable and contentious [47].”
BackgroundSelf-citationA Human-AI Collaborative Approach for Clinical Decision Making on Rehabilitation Assessment
Lee, M. H., Siewiorek, D. P., Smailagic, A., Bernardino, A., & Bermúdez i Badia, S. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445472 · Cited by 137 · cited 1×
“These issues have motivated studies that aim to evaluate and improve the humanAI collaborative system in health care [35, 40, 53], often drawing on insights from interpretable AI [52, 67, 85].”
BackgroundThe Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123 · cited 1×
“However, contrary to prior work showing negative efects of task expertise and AI familiarity on acceptance of AI recommendations [7, 25, 31, 40], the behavior patterns we observed did not appear correlated to seniority or experience level.”
ResultSuresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445088 · Cited by 117 · cited 1×
“The design of explainable and interpretable ML-based tools has become a major focus of research in the HCI community [26, 76, 90].”
BackgroundCalisto, F. M., Santiago, C., Nunes, N., & Nascimento, J. C. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102607 · Cited by 117 · cited 2×
“either agreement or disagreement with a diagnosis) [14, 16, 35, 45, 80, 83].”
BackgroundModeling adoption of intelligent agents in medical imaging
Calisto, F. M., Nunes, N., & Nascimento, J. C. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102922 · Cited by 111
Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 103
The role of domain expertise in trusting and following explainable AI decision support systems
Bayer, S., Gimpel, H., & Markgraf, M. (2022) · Journal of Decision Systems · doi:10.1080/12460125.2021.1958505 · Cited by 103 · cited 1×
“However, contrary to prior work showing negative effects of task expertise and AI familiarity on acceptance of AI recommendations [7, 25, 31, 40], the behavior patterns we observed did not appear correlated to seniority or experience level.”
ResultOperationalizing Human-Centered Perspectives in Explainable AI
Ehsan, U., Wintersberger, P., Liao, Q. V., Mara, M., Streit, M., Wachter, S., Riener, A., & Riedl, M. O. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3441342 · Cited by 97 · cited 1×
“The design of explainable and interpretable ML-based tools has become a major focus of research in the HCI community [26, 76, 90].”
BackgroundExplainable Artificial Intelligence Approaches: A Survey
Islam, S. R., Eberle, W., Ghafoor, S., & Ahmed, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.09429 · Cited by 88 · cited 1×
“While early eforts in explainable AI (XAI) focused on feature-based explanations, current conceptions of interpretability comprise a wider range of techniques, including uncertainty and confdence metrics [65], nearest-neighbors [39], and counterfactuals [91].”
BackgroundRe-focusing explainability in medicine
Arbelaez Ossa, L., Starke, G., Lorenzini, G., Vogt, J. E., Shaw, D. M., & Elger, B. S. (2022) · DIGITAL HEALTH · doi:10.1177/20552076221074488 · Cited by 87 · cited 1×
“Our study also examined the efects of model explainability, an ongoing area of debate in AI-based CDS research [2, 3, 32], on participants’ perceptions and behaviors using the AI.”
BackgroundNational Performance on the Medicare SEP-1 Sepsis Quality Measure
Barbash, I. J., Davis, B., & Kahn, J. M. (2019) · Critical Care Medicine · doi:10.1097/ccm.0000000000003613 · Cited by 86 · cited 1×
“As such, recommendations face signifcant challenges in translation to wider clinical practice [75], resulting in substantial variability in care practices [6] and continued high mortality levels [74].”
BackgroundSelf-citationExamining the effect of explanation on satisfaction and trust in AI diagnostic systems
Alam, L., & Mueller, S. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01542-6 · Cited by 81 · cited 3×
“A mixed-methods analysis of the think-aloud transcripts and structured decision responses showed that explanations improved clinicians’ perceptions of the AI’s usefulness and made them more confdent in their own decisions, a fnding that is consistent with prior literature [1, 80, 85].”
BackgroundResultTowards a Science of Human-AI Decision Making: A Survey of Empirical Studies
Lai, V., Chen, C., Liao, Q. V., Smith-Renner, A., & Tan, C. (2021) · arXiv · doi:10.48550/arxiv.2112.11471 · Cited by 75 · cited 2×
“In order to cultivate an appropriate level of reliance, interpretable AI systems must account for decision-maker characteristics, task complexity, AI performance, and other factors in ways that are not yet fully understood [52, 57, 67].”
BackgroundHow good is the evidence to support primary care practice?
Ebell, M. H., Sokol, R. G., Lee, A., Simons, C., & Early, J. (2017) · BMJ Evidence-Based Medicine · doi:10.1136/ebmed-2017-110704 · Cited by 70
State Sepsis Mandates — A New Era for Regulation of Hospital Quality
Hershey, T. B., & Kahn, J. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1611928 · Cited by 65
Self-citationZytek, A., Liu, D., Vaithianathan, R., & Veeramachaneni, K. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114864 · Cited by 62 · cited 1×
“While early eforts in explainable AI (XAI) focused on feature-based explanations, current conceptions of interpretability comprise a wider range of techniques, including uncertainty and confdence metrics [65], nearest-neighbors [39], and counterfactuals [91].”
BackgroundImproving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learning
Peng, X., Ding, Y., Wihl, D., Gottesman, O., Komorowski, M., Lehman, L.-W. H., Ross, A. S., Faisal, A. A., & Doshi‐Velez, F. (2019) · PubMed · doi:10.48550/arxiv.1901.04670 · Cited by 62 · cited 1×
“Additionally, more recently-developed techniques using deep neural networks may improve on the accuracy of the AI Clinician [29, 61, 89].”
MethodExplainable Reinforcement Learning through a Causal Lens
Madumal, P., Miller, T., Sonenberg, L., & Vetere, F. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i03.5631 · Cited by 58
Sepsis Bundles and Compliance With Clinical Guidelines
Stoneking, L., Denninghoff, K. R., DeLuca, L., Keim, S. M., & Munger, B. S. (2011) · Journal of Intensive Care Medicine · doi:10.1177/0885066610387988 · Cited by 47 · cited 1×
“As such, recommendations face significant challenges in translation to wider clinical practice [75], resulting in substantial variability in care practices [6] and continued high mortality levels [74].”
BackgroundFluid resuscitation during early sepsis: a need for individualization
Jozwiak, M., Hamzaoui, O., Monnet, X., & Teboul, J. (2018) · Minerva Anestesiologica · doi:10.23736/s0375-9393.18.12422-9 · Cited by 45 · cited 1×
“It is impossible to know if the AI’s recommendation to give vasopressors was a better decision, although some evidence shows that early administration of vasopressors could beneft patients [72] and expert opinion increasingly emphasizes vasopressors over fuid administration [44].”
BackgroundAmbiguity-aware AI Assistants for Medical Data Analysis
Schaekermann, M., Beaton, G., Sanoubari, E., Lim, A., Larson, K., & Law, E. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376506 · Cited by 44 · cited 1×
“While early efforts in explainable AI (XAI) focused on feature-based explanations, current conceptions of interpretability comprise a wider range of techniques, including uncertainty and confidence metrics [65], nearest-neighbors [39], and counterfactuals [91].”
BackgroundSadasivuni, S., Saha, M., Bhatia, N. K., Banerjee, I., & Sanyal, A. (2022) · Scientific Reports · doi:10.1038/s41598-022-09712-w · Cited by 42
A Review of Predictive Analytics Solutions for Sepsis Patients
Teng, A. K., & Wilcox, A. B. (2020) · Applied Clinical Informatics · doi:10.1055/s-0040-1710525 · Cited by 41 · cited 1×
“Multiple machine learning algorithms exist for mining hospital electronic health record data to identify patients with sepsis [58, 64, 77].”
BackgroundAn Appraisal of the Evidence Underlying Performance Measures for Community-acquired Pneumonia
Wilson, K. C., & Schünemann, H. J. (2011) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.201009-1451pp · Cited by 40 · cited 1×
“In these systems, the “best” decisions are often difcult to identify, either due to a lack of clinical evidence [22, 86] or due to expert disagreement on the best course of action [24, 63].”
BackgroundClinical decision support for therapeutic decision-making in cancer: a systematic review
Beauchemin, M., Murray, M. T., Sung, L., Hershman, D. L., Weng, C., & Schnall, R. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.07.019 · Cited by 38 · cited 1×
“In contrast to diagnoses, which are relatively straightforward to learn from historical data, treatment recommendations in CDS have predominantly been derived from clinical best-practice guidelines rather than AI [8, 43].”
ResultDeep Inverse Reinforcement Learning for Sepsis Treatment
Yu, C., Ren, G., & Liu, J. (2019) · 2019 IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi.2019.8904645 · Cited by 38 · cited 1×
“Additionally, more recently-developed techniques using deep neural networks may improve on the accuracy of the AI Clinician [29, 61, 89].”
MethodKaltenhauser, A., Rheinstädter, V., Butz, A. M., & Wallach, D. (2020) · Designing Interactive Systems Conference · doi:10.1145/3357236.3395436 · Cited by 38 · cited 3×
“standing decision-making without an AI rather than the infuence of AI on treatment decisions [46].”
BackgroundDobler, C. C., Morrow, A. S., & Kamath, C. C. (2018) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2018-111074 · Cited by 37 · cited 1×
“Explanations can also interact with reasoning fallacies such as confirmation and availability bias [21, 45], but mitigating these effects requires knowledge of a normatively correct reasoning process [84] that may not always be available.”
BackgroundJones, B. E., Collingridge, D. S., Vines, C. G., Post, H., Holmen, J. R., Allen, T., Haug, P. J., Weir, C. R., & Dean, N. C. (2019) · Applied Clinical Informatics · doi:10.1055/s-0038-1676587 · Cited by 37 · cited 1×
“In contrast to diagnoses, which are relatively straightforward to learn from historical data, treatment recommendations in CDS have predominantly been derived from clinical best-practice guidelines rather than AI [8, 43].”
ResultUnderstanding Why We Agree on the Evidence but Disagree on the Medicine
Rubenfeld, G. D. (2001) · Respiratory Care · doi:10.1177/002013240104601208 · Cited by 36 · cited 1×
“In these systems, the “best” decisions are often difcult to identify, either due to a lack of clinical evidence [22, 86] or due to expert disagreement on the best course of action [24, 63].”
BackgroundClinicalVis: Supporting Clinical Task-Focused Design Evaluation
Ghassemi, M., Pushkarna, M., Wexler, J., Johnson, J., & Varghese, P. (2018) · arXiv · doi:10.48550/arxiv.1810.05798 · Cited by 26 · cited 2×
“An initial prototype was created using inspiration from prior literature, notably ClinicalVis [33].”
MethodResultReliable Post hoc Explanations: Modeling Uncertainty in Explainability
Slack, D., Hilgard, S., Singh, S. K., & Lakkaraju, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05030 · Cited by 23 · cited 1×
“Model explanations themselves can be prone to issues such as over-sensitivity to input values or giving seemingly-sensible explanations for incorrect predictions [73].”
BackgroundEarly Prediction of Sepsis in the ICU using Machine Learning: A Systematic ReviewPreprint
Moor, M., Rieck, B., Horn, M., et al. (2020) · medRxiv · doi:10.1101/2020.08.31.20185207 · Cited by 20
Trust and Reliance in XAI -- Distinguishing Between Attitudinal and Behavioral Measures
Scharowski, N., Perrig, S. A. C., von Felten, N., & Brühlmann, F. (2022) · arXiv · doi:10.48550/arxiv.2203.12318 · Cited by 18 · cited 1×
“Rather than evaluating decisionmaking along a single axis of quality, we used a combination of behavioral and attitudinal measures [66] to understand how an imperfect AI system would afect its users within a high-stakes”
MethodCurrent Sepsis Mandates Are Overly Prescriptive, and Some Aspects May Be Harmful
Klompas, M., & Rhee, C. (2018) · Critical Care Medicine · doi:10.1097/ccm.0000000000003579 · Cited by 17 · cited 1×
“However, broad best-practice guidelines are of limited utility at the bedside because their recommendations fail to account for patient-level variation and the interaction between multiple variables over time [49].”
BackgroundGeneralizing Off-Policy Evaluation From a Causal Perspective For Sequential Decision-MakingPreprint
Parbhoo, S., Joshi, S. D., & Doshi‐Velez, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.08262 · Cited by 8 · cited 1×
“This complex reasoning task is well-known in causal modeling [60] and has been described in early-stage design studies in health care [46].”
BackgroundMansoori, J. N., Clark, B. J., Havranek, E. P., & Douglas, I. S. (2022) · MDM Policy & Practice · doi:10.1177/23814683221099454 · Cited by 3 · cited 1×
“Although the recommendation included specifc dosages of IV fuids and vasopressors, we limited participants’ choice set to (up to) three options in an efort to capture clinicians’ frst-order decisionmaking process, and to better replicate the way clinicians make resuscitation decisions at the bedside [56].”
BackgroundHuman-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-MakingPreprint
Cai, C. J., Reif, E., Hegde, N., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02960 · cited 1×
“Taking an alternative strategy, some AI systems instead strive to provide clinicians with useful non-prescriptive information, such as highlighting informative parts of a medical image [27], displaying information from similar historical cases [13], or identifying patients at high risk of future deterioriation [35, 68…”
BackgroundThe Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.13509 · cited 1×
“However, contrary to prior work showing negative efects of task expertise and AI familiarity on acceptance of AI recommendations [7, 25, 31, 40], the behavior patterns we observed did not appear correlated to seniority or experience level.”
ResultClinical Decision Making: Theory vs Practice-Reply
Eddy, D. M. (1990) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1990.03450120045020
Further Appraisal of the Evidence Underlying Performance Measures for Community-acquired Pneumonia
Wilson, K. C., & Schünemann, H. J. (2011) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/ajrccm.184.11.1314a
Kawakami, A., Sivaraman, V., Cheng, H.-F., Stapleton, L., Cheng, Y., Qing, D., Perer, A., Wu, Z. S., Zhu, H., & Holstein, K. (2022) · arXiv · doi:10.48550/arxiv.2204.02310 · cited 1×
“While expert consensus can serve as a useful proxy for the ground truth [40], the accuracy of a real-world decision is often fundamentally unknowable and contentious [47].”
BackgroundSelf-citation
Research works printed but not linked: 4
- Joseph Futoma, Anthony Lin, Mark Sendak, Armando Bedoya, Meredith Clement, Cara O’Brien, and Katherine Heller. 2018. Learning to Treat Sepsis with Multi-Output Gaussian Process Deep Recurrent Q-Networks. ICLR 2018 Conference Blind Submission2017 (2018), 1–10. https://openreview.net/pdf?id=SyxCqGbRZ
- 10.1097/CCM.0b013e318250a887.Randomized · doi:10.1097/ccm.0b013e318250a887.randomized
- 10.1038/s41591-018-0213-5 · DOI · Paper page
- Nirav R Shah and Thomas H Lee. 2019. What AI means for doctors and doctoring. NEJM Catalyst 5, 5 (2019).
Cited by
Tun, H. M., Rahman, H. A., Naing, L., & Malik, O. A. (2025) · Journal of Medical Internet Research · doi:10.2196/69678 · Cited by 210
“Sivaraman et al [28] and Jacobs et al [5] discussed the important role of a sociotechnical lens in designing AI-CDSS, emphasizing the need to integrate environmental and social factors into system development.”
A literature review on applications of explainable artificial intelligence (XAI)
Kalasampath, K., Spoorthi, K. N., Sajeev, S., Kuppa, S. S., Ajay, K., & Maruthamuthu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3546681 · Cited by 180
“They highlight the importance of establishing standardized metrics to evaluate the effectiveness of XAI models and fostering multidisciplinary collaboration [110], [119].”
Miller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172
“…trust calibration people tend to either under-rely on tools, meaning they have no effect on decision making, or they over-rely on tools [Buçinca et al., 2021, Gajos and Mamykina, 2022, Sivaraman et al., 2023], likely because providing recommendations makes them fixate on that recommendation.”
BackgroundMethodInfluentialAI for Decision Support: Balancing Accuracy, Transparency, and Trust Across Sectors
Kővári, A. (2024) · Information · doi:10.3390/info15110725 · Cited by 213
Explainable AI improves task performance in human–AI collaboration
Senoner, J., Schallmoser, S., Kratzwald, B., Feuerriegel, S., & Netland, T. H. (2024) · Scientific Reports · doi:10.1038/s41598-024-82501-9 · Cited by 200
Metadata
- Authors
- Sivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., & Perer, A.
- Affiliations
- Carnegie Mellon University (US)
- University of Pittsburgh (US)
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-18
- OpenAlex
- W4366547677
- Cited by
- 141
- References
- 67
- Citations by year
- 2026: 45 · 2025: 48 · 2024: 34 · 2023: 14
- FWCI
- 25.45
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Center for Machine Learning and Health, School of Computer Science, Carnegie Mellon University
- National Institutes of Health (R35HL144804)
- National Science Foundation (DGE2140739)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- artificial intelligence in healthcare, AI-assisted decision-making, decision support interface, sepsis, trust in AI, clinical decision-making, AI adoption barriers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.