CoChat

Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care

Open access
Sivaraman, Venkatesh, Bukowski, Leigh A, Levin, Joel, Kahn, Jeremy M., Perer, Adam · 2023 · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

The DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index; none of the 31 sources its claims cite is checked for support yet.

Abstract

Artificial intelligence (AI) in healthcare has the potential to improve patient outcomes, but clinician acceptance remains a critical barrier. We developed a novel decision support interface that provides interpretable treatment recommendations for sepsis, a life-threatening condition in which decisional uncertainty is common, treatment practices vary widely, and poor outcomes can occur even with optimal decisions. This system formed the basis of a mixed-methods study in which 24 intensive care clinicians made AI-assisted decisions on real patient cases. We found that explanations generally increased confidence in the AI, but concordance with specific recommendations varied beyond the binary acceptance or rejection described in prior work. Although clinicians sometimes ignored or trusted the AI, they also often prioritized aspects of the recommendations to follow, reject, or delay in a process we term “negotiation.” These results reveal novel barriers to adoption of treatment-focused AI tools and suggest ways to better support differing clinician perspectives.

Overview

CoChat Summary

TL;DR

In a mixed-methods think-aloud study with 24 ICU clinicians using an AI Clinician-based sepsis treatment CDS, explanations increased perceived usefulness and confidence in the AI but did not change overall binary concordance with recommendations. Clinicians often engaged in partial reliance—“negotiation”—rather than simply ignoring or trusting the AI, revealing barriers for treatment-focused AI adoption.

Key findings

  • Explanations improved perceptions: Feature Explanation was rated more useful than Text Only (Δ = 0.83, 95% CI [0.24, 1.43], p = 0.018) and had a more positive effect on confidence (Δ = 1.08, 95% CI [0.51, 1.66], p < 0.001); Alternative Treatments showed directionally similar benefits.
  • Concordance with the AI did not vary meaningfully by visualization condition: clinicians chose the same treatment as the AI about 42% of the time across AI conditions, only slightly above the 33% base rate without seeing the recommendation. Full concordance with the majority attending decision averaged 50% across AI conditions versus 63% in the No AI condition.
  • Four behavior patterns emerged: Ignore (7 participants, 21 decisions), Negotiate (12 participants, 36 decisions), Consider (3 participants, 9 decisions), and Trust/Rely (2 participants). Negotiate participants rated recommendations more useful (M = 4.6, SD = 1.27) than other groups (M = 2.8, SD = 1.79).
  • Explanations increased perceived case difficulty: No AI was rated less challenging than Alternative Treatments (Δ = 1.08, 95% CI [0.46, 1.71], p = 0.003) and directionally less challenging than Feature Explanation (Δ = 0.79, 95% CI [0.14, 1.45], p = 0.09).

Methods

  • Mixed-methods think-aloud study with 24 practicing ICU clinicians (attending physicians, advanced practice providers, and critical care fellows) from a large multihospital academic health system. Each participant made AI-assisted treatment decisions for four real de-identified MIMIC-IV sepsis cases, with four visualization conditions presented in fixed order: No AI, Text Only, Feature Explanation, and Alternative Treatments.
  • The CDS was built on a replicated AI Clinician reinforcement learning model trained on 18,143 MIMIC-IV sepsis patients, using 750 patient states and 25 treatment actions (5 IV fluid levels and 5 vasopressor doses binned by quantiles). The displayed model had a policy value of 83.8, compared with original reported values of 80–90. Cases were selected where the AI recommendation substantially differed from historical clinician actions.
  • Quantitative analysis used OLS regression for 7-point Likert outcomes and logistic regression for concordance, with clustered standard errors and Holm-Bonferroni post hoc adjustment. Qualitative analysis coded 12.6 hours of think-aloud transcripts using 23 codes, plus open coding and affinity diagramming.

Limitations

  • Stated: case selection may have skewed observations toward particular cases and recommendations; patient depictions were limited to structured MIMIC-IV data; domain-specific explainable RL techniques were not used; SHAP explanations focused only on state clustering, limiting their potential usefulness.
  • Stated: the study design prioritized rich think-aloud data, so statistical significance of some quantitative results, especially concordance, could not be assessed; the academic hospital setting may limit generalizability.
  • Evident from the text: visualization conditions were presented in a fixed order, which may introduce order or learning effects; and because there is no objective ground truth for treatment decisions, concordance with the AI is not a direct measure of decision accuracy.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3544548.3581075

Cites

91 of 91 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 0 Not valid · 17 Not checked · 80 Own results and interpretations

44 more claims

Create a free account to see all 97 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 17
  • The quote only partly states this claim. 4
  • Probably not a substantial claim. 30

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

26 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 91 references

Research works printed but not linked: 4

  1. Joseph Futoma, Anthony Lin, Mark Sendak, Armando Bedoya, Meredith Clement, Cara O’Brien, and Katherine Heller. 2018. Learning to Treat Sepsis with Multi-Output Gaussian Process Deep Recurrent Q-Networks. ICLR 2018 Conference Blind Submission2017 (2018), 1–10. https://openreview.net/pdf?id=SyxCqGbRZ
  2. 10.1097/CCM.0b013e318250a887.Randomized · doi:10.1097/ccm.0b013e318250a887.randomized
  3. 10.1038/s41591-018-0213-5 · DOI · Paper page
  4. Nirav R Shah and Thomas H Lee. 2019. What AI means for doctors and doctoring. NEJM Catalyst 5, 5 (2019).

Cited by

See all 143

Metadata

Authors
Sivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., & Perer, A.
Affiliations
  • Carnegie Mellon University (US)
  • University of Pittsburgh (US)
Venue
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2023-04-19
Pages
1-18
OpenAlex
W4366547677
Cited by
141
References
67
Citations by year
2026: 45 · 2025: 48 · 2024: 34 · 2023: 14
FWCI
25.45
Type
conference paper
Language
en
Open access
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, other-oa)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Center for Machine Learning and Health, School of Computer Science, Carnegie Mellon University
  • National Institutes of Health (R35HL144804)
  • National Science Foundation (DGE2140739)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
artificial intelligence in healthcare, AI-assisted decision-making, decision support interface, sepsis, trust in AI, clinical decision-making, AI adoption barriers

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care. https://app.cochat.ai/papers/10.1145/3544548.3581075
The paper
Sivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., & Perer, A. (2023). Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3544548.3581075

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.