Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care
Open accessThe DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index; none of the 31 sources its claims cite is checked for support yet.
Abstract
Artificial intelligence (AI) in healthcare has the potential to improve patient outcomes, but clinician acceptance remains a critical barrier. We developed a novel decision support interface that provides interpretable treatment recommendations for sepsis, a life-threatening condition in which decisional uncertainty is common, treatment practices vary widely, and poor outcomes can occur even with optimal decisions. This system formed the basis of a mixed-methods study in which 24 intensive care clinicians made AI-assisted decisions on real patient cases. We found that explanations generally increased confidence in the AI, but concordance with specific recommendations varied beyond the binary acceptance or rejection described in prior work. Although clinicians sometimes ignored or trusted the AI, they also often prioritized aspects of the recommendations to follow, reject, or delay in a process we term “negotiation.” These results reveal novel barriers to adoption of treatment-focused AI tools and suggest ways to better support differing clinician perspectives.
Overview
CoChat Summary
TL;DR
In a mixed-methods think-aloud study with 24 ICU clinicians using an AI Clinician-based sepsis treatment CDS, explanations increased perceived usefulness and confidence in the AI but did not change overall binary concordance with recommendations. Clinicians often engaged in partial reliance—“negotiation”—rather than simply ignoring or trusting the AI, revealing barriers for treatment-focused AI adoption.
Key findings
- Explanations improved perceptions: Feature Explanation was rated more useful than Text Only (Δ = 0.83, 95% CI [0.24, 1.43], p = 0.018) and had a more positive effect on confidence (Δ = 1.08, 95% CI [0.51, 1.66], p < 0.001); Alternative Treatments showed directionally similar benefits.
- Concordance with the AI did not vary meaningfully by visualization condition: clinicians chose the same treatment as the AI about 42% of the time across AI conditions, only slightly above the 33% base rate without seeing the recommendation. Full concordance with the majority attending decision averaged 50% across AI conditions versus 63% in the No AI condition.
- Four behavior patterns emerged: Ignore (7 participants, 21 decisions), Negotiate (12 participants, 36 decisions), Consider (3 participants, 9 decisions), and Trust/Rely (2 participants). Negotiate participants rated recommendations more useful (M = 4.6, SD = 1.27) than other groups (M = 2.8, SD = 1.79).
- Explanations increased perceived case difficulty: No AI was rated less challenging than Alternative Treatments (Δ = 1.08, 95% CI [0.46, 1.71], p = 0.003) and directionally less challenging than Feature Explanation (Δ = 0.79, 95% CI [0.14, 1.45], p = 0.09).
Methods
- Mixed-methods think-aloud study with 24 practicing ICU clinicians (attending physicians, advanced practice providers, and critical care fellows) from a large multihospital academic health system. Each participant made AI-assisted treatment decisions for four real de-identified MIMIC-IV sepsis cases, with four visualization conditions presented in fixed order: No AI, Text Only, Feature Explanation, and Alternative Treatments.
- The CDS was built on a replicated AI Clinician reinforcement learning model trained on 18,143 MIMIC-IV sepsis patients, using 750 patient states and 25 treatment actions (5 IV fluid levels and 5 vasopressor doses binned by quantiles). The displayed model had a policy value of 83.8, compared with original reported values of 80–90. Cases were selected where the AI recommendation substantially differed from historical clinician actions.
- Quantitative analysis used OLS regression for 7-point Likert outcomes and logistic regression for concordance, with clustered standard errors and Holm-Bonferroni post hoc adjustment. Qualitative analysis coded 12.6 hours of think-aloud transcripts using 23 codes, plus open coding and affinity diagramming.
Limitations
- Stated: case selection may have skewed observations toward particular cases and recommendations; patient depictions were limited to structured MIMIC-IV data; domain-specific explainable RL techniques were not used; SHAP explanations focused only on state clustering, limiting their potential usefulness.
- Stated: the study design prioritized rich think-aloud data, so statistical significance of some quantitative results, especially concordance, could not be assessed; the academic hospital setting may limit generalizability.
- Evident from the text: visualization conditions were presented in a fixed order, which may introduce order or learning effects; and because there is no objective ground truth for treatment decisions, concordance with the AI is not a direct measure of decision accuracy.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 17 Not checked · 80 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“two participants were infuenced by the AI for at least part of their decision in every decision they made”
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > 5.2.4
plainstates a causen = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants perceived the cases as being signifcantly less challenging in the No AI condition than in the Alternative Treatments condition”
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > Efect of AI on confdence.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants' confdence in their treatment choices was not signifcantly diferent across Visualization conditions”
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > Efect of AI on confdence.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Visualization condition afected how participants rated the AI's impact on their confdence”
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > Efect of AI on confdence.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In 12/36 decisions, participants used their perception of the severity of the patient's sepsis to decide how much to reconsider their treatment plan”
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > 5.2.2 Negotiate:
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
44 more claims
Create a free account to see all 97 claims and every source verdict.
- It may state two separate claims as one. 17
- The quote only partly states this claim. 4
- Probably not a substantial claim. 30
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Trust was driven more by evidence of external validation than interface features or explanation types.”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [62]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This study directly investigates clinician interaction, acceptance, and trust-including 'negotiation' behaviors-with an interpretable AI-CDSS for sepsis treatment [62].”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [62]
The cited source says what the claim attributes to it.
In the citing paper
“However, people find it difficult to correctly calibrate their trust in a decision aid [10,13,21,37].”
From Explainable AI is Dead, Long Live Explainable AI!: Hypothesis-driven Decision Support using Evaluative AI (2023) · cites this paper as [37]
The cited source says what the claim attributes to it.
In the citing paper
“Second, instead of full consensus or complete divergence, human and AI decision rationales often exhibit partial alignment [113, 128].”
From Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making (2025) · cites this paper as [113]
In the citing paper
“User expertise also plays a role: more knowledgeable clinicians are often better able to calibrate their trust in an AI system appropriately [58,62].”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [62]
26 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > 6.2 Validating that AI-Based Decision Support Improves Outcomes
“the recent guidance by the U.S. Food and Drug Administration that treatment decision support systems such as the AI Clinician should be regulated as medical devices [81].”
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > References
“82. U.S. Food and Drug Administration. 2022. Clinical Decision Support Software. Technical Report. 1-26 pages. https://www.fda.gov/media/109618/download”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > 6 DISCUSSION
“for these participants the CDS only served to lower their confdence in their decision-making and increase the perceived difculty of the case.”
§ Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care > 5.2.1 Ignore: Participant makes own decisions.
“In these cases the AI served to either confrm the initial assessment-"made me feel better about that decision" (P5)-or, more commonly, to induce doubt when the recommendation was discordant (P6, P7, P17, P18).”
Citations
Cites
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569 · cited 1×
“For each of the 750 possible states, we trained an XGBoost classifer [19] to predict whether a patient was in that state or not.”
MethodA Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 5×
“In addition to the textual recommendation, participants were shown a SHAP feature attribution chart explaining how the patient’s state was determined (Fig.”
MethodInfluentialHigh-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · cited 1×
“The AI Clinician publication is widely known and highly infuential in both the critical care and CDS communities [79, 88].”
BackgroundSurviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock 2021.
Evans, L., Rhodes, A., Alhazzani, W., Antonelli, M., Coopersmith, C. M., French, C., Machado, F. R., Mcintyre, L., Ostermann, M., Prescott, H. C., et al. (2021) · Critical Care Medicine · doi:10.1097/ccm.0000000000005337 · Cited by 2,935
Shankar-Hari, M., Phillips, G. S., Levy, M. L., Seymour, C. W., Liu, V. X., Deutschman, C. S., Angus, D. C., Rubenfeld, G. D., Singer, M., & for the Sepsis Definitions Task Force (2016) · JAMA · doi:10.1001/jama.2016.0289 · Cited by 2,424
Cecconi, M., Evans, L., Levy, M., & Rhodes, A. (2018) · The Lancet · doi:10.1016/s0140-6736(18)30696-2 · Cited by 2,254
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724 · cited 3×
“However, none of these methods appear to work universally across contexts, particularly in health care; in some domains, they may infate or undermine confdence [31, 40, 90], while in others they may be poorly-aligned with human decision-making processes [2, 32].”
BackgroundDeep learning-enabled medical computer vision
Esteva, A., Chou, K., Yeung, S., Naik, N., Madani, A., Mottaghi, A., Liu, Y., Topol, E., Dean, J., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00376-2 · Cited by 1,406 · cited 1×
“Taking an alternative strategy, some AI systems instead strive to provide clinicians with useful non-prescriptive information, such as highlighting informative parts of a medical image [27], displaying information from similar historical cases [13], or identifying patients at high risk of future deterioriation [35, 68…”
BackgroundKomorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., & Faisal, A. A. (2018) · Nature Medicine · doi:10.1038/s41591-018-0213-5 · Cited by 1,290 · cited 4×
“[50] but that has not been prospectively evaluated in a clinical setting.”
BackgroundMethodInfluentialBuçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Research works printed but not linked: 4
- Joseph Futoma, Anthony Lin, Mark Sendak, Armando Bedoya, Meredith Clement, Cara O’Brien, and Katherine Heller. 2018. Learning to Treat Sepsis with Multi-Output Gaussian Process Deep Recurrent Q-Networks. ICLR 2018 Conference Blind Submission2017 (2018), 1–10. https://openreview.net/pdf?id=SyxCqGbRZ
- 10.1097/CCM.0b013e318250a887.Randomized · doi:10.1097/ccm.0b013e318250a887.randomized
- 10.1038/s41591-018-0213-5 · DOI · Paper page
- Nirav R Shah and Thomas H Lee. 2019. What AI means for doctors and doctoring. NEJM Catalyst 5, 5 (2019).
Cited by
Tun, H. M., Rahman, H. A., Naing, L., & Malik, O. A. (2025) · Journal of Medical Internet Research · doi:10.2196/69678 · Cited by 210
“Sivaraman et al [28] and Jacobs et al [5] discussed the important role of a sociotechnical lens in designing AI-CDSS, emphasizing the need to integrate environmental and social factors into system development.”
A literature review on applications of explainable artificial intelligence (XAI)
Kalasampath, K., Spoorthi, K. N., Sajeev, S., Kuppa, S. S., Ajay, K., & Maruthamuthu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3546681 · Cited by 180
“They highlight the importance of establishing standardized metrics to evaluate the effectiveness of XAI models and fostering multidisciplinary collaboration [110], [119].”
Miller, T. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594001 · Cited by 172
“…trust calibration people tend to either under-rely on tools, meaning they have no effect on decision making, or they over-rely on tools [Buçinca et al., 2021, Gajos and Mamykina, 2022, Sivaraman et al., 2023], likely because providing recommendations makes them fixate on that recommendation.”
BackgroundMethodInfluentialAI for Decision Support: Balancing Accuracy, Transparency, and Trust Across Sectors
Kővári, A. (2024) · Information · doi:10.3390/info15110725 · Cited by 213
Explainable AI improves task performance in human–AI collaboration
Senoner, J., Schallmoser, S., Kratzwald, B., Feuerriegel, S., & Netland, T. H. (2024) · Scientific Reports · doi:10.1038/s41598-024-82501-9 · Cited by 199
Metadata
- Authors
- Sivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., & Perer, A.
- Affiliations
- Carnegie Mellon University (US)
- University of Pittsburgh (US)
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-18
- OpenAlex
- W4366547677
- Cited by
- 141
- References
- 67
- Citations by year
- 2026: 45 · 2025: 48 · 2024: 34 · 2023: 14
- FWCI
- 25.45
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Center for Machine Learning and Health, School of Computer Science, Carnegie Mellon University
- National Institutes of Health (R35HL144804)
- National Science Foundation (DGE2140739)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- artificial intelligence in healthcare, AI-assisted decision-making, decision support interface, sepsis, trust in AI, clinical decision-making, AI adoption barriers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care. https://app.cochat.ai/papers/10.1145/3544548.3581075
- The paper
- Sivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., & Perer, A. (2023). Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3544548.3581075
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.