Cited by 99Artificial Intelligence in Healthcare and EducationMachine Learning in HealthcareElectronic Health Records Systems

Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosis

Open access
Zhang, Shao, Yu, Jianing, Xu, Xuhai, Yin, Changchang, Lu, Yuxuan, Yao, Bingsheng, Tory, Melanie, Padilla, Lace M. +3 more·2024·Proceedings of the CHI Conference on Human Factors in Computing Systems

Abstract

Today's AI systems for medical decision support often succeed on benchmark datasets in research papers but fail in real-world deployment. This work focuses on the decision making of sepsis, an acute life-threatening systematic infection that requires an early diagnosis with high uncertainty from the clinician. Our aim is to explore the design requirements for AI systems that can support clinical experts in making better decisions for the early diagnosis of sepsis. The study begins with a formative study investigating why clinical experts abandon an existing AI-powered Sepsis predictive module in their electrical health record (EHR) system. We argue that a human-centered AI system needs to support human experts in the intermediate stages of a medical decision-making process (e.g., generating hypotheses or gathering data), instead of focusing only on the final decision. Therefore, we build SepsisLab based on a state-of-the-art AI algorithm and extend it to predict the future projection of sepsis development, visualize the prediction uncertainty, and propose actionable suggestions (i.e., which additional laboratory tests can be collected) to reduce such uncertainty. Through heuristic evaluation with six clinicians using our prototype system, we demonstrate that SepsisLab enables a promising human-AI collaboration paradigm for the future of AI-assisted sepsis diagnosis and other high-stakes medical decision making.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3613904.3642343

Cites

96 of 96 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Information-driven support may provide users with more agency and autonomy than recommendation-driven support.

    “In addition to supporting decision making, information-driven support may provide users with more agency and autonomy than recommendation-driven support [82, 83].”

    From To Recommend or Not to Recommend: Designing and Evaluating AI-Enabled Decision Support for Time-Critical Medical Events (2025) · cites this paper as [82]

    Section 3.2.5 (page unavailable) says AI output that suggests a final diagnosis can challenge clinicians’ authority. Section 6.1 (page unavailable) argues that information and support at intermediate stages can preserve clinicians’ role as final decision-makers. But the proposed system also gives laboratory-test recommendations, so the paper does not support a general contrast between information-driven and recommendation-driven support. (LLM, conf=0.84)

    Experimental — accuracy not yet measured
  • Information-driven support could help providers evaluate potential diagnoses while avoiding issues commonly found with recommendations, such as over-reliance and reduced agency.

    “Information-driven support could help providers evaluate potential diagnoses, while avoiding issues commonly found with recommendations, such as over-reliance and reduced agency [45, 82, 83].”

    From To Recommend or Not to Recommend: Designing and Evaluating AI-Enabled Decision Support for Time-Critical Medical Events (2025) · cites this paper as [82]

    Sections 5.2.2 and 6.1 say that information supporting hypothesis testing can help clinicians make decisions while preserving their role as final decision-makers. But Section 6.3 says over-dependence remains a potential risk of the proposed approach and that avoiding undue reliance is an open design question; the paper does not establish that information-driven support avoids it. Page numbers were unavailable in our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured
  • Recent research in human-AI interaction has proposed a paradigm shift from recommendation-driven support to information-driven support.

    “However, recent research in human-AI interaction has proposed a paradigm shift from recommendation-driven support to information-driven support, which provides users with information synthesis to facilitate hypothesis testing [27, 45, 82, 83].”
  • Some intensive care and emergency medicine physicians preferred receiving information synthesis from AI-enabled systems to help them generate and test hypotheses instead of risk prediction scores that could challenge their authority.

    “In one study, some intensive care and emergency medicine physicians preferred receiving information synthesis from AI-enabled systems to help them generate and test hypotheses instead of risk prediction scores that could challenge their authority [82].”
  • Prompting critical thinking or "provocation" without challenging provider expertise is a challenge in designing AI support for clinical settings.

    “Another challenge in designing AI support for clinical settings is prompting critical thinking or “provocation” without challenging provider expertise [82].”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 12

  1. Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. 2016. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. Advances in neural information processing systems 29 (2016).
  2. Leo A Goodman. 1961. Snowball sampling. The annals of mathematical statistics (1961), 148-170.
  3. Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. 2021. Towards a science of human-ai decision making: a survey of empirical studies. arXiv preprint arXiv:2112.11471 (2021). · arXiv
  4. Robyn Longhurst. 2003. Semi-structured interviews and focus groups. Key methods in geography 3, 2 (2003), 143-156.
  5. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gre- gory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc.
  6. Mark Sendak, Madeleine Clare Elish, Michael Gao, Joseph Futoma, William Ratlif, Marshall Nichols, Armando Bedoya, Suresh Balu, and Cara O'Brien. 2020. " The human body is a black box" supporting clinical decision-making with deep learning. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 99-109.
  7. Nirav R Shah and Thomas H Lee. 2019. What AI means for doctors and doctoring. NEJM Catalyst 5, 5 (2019).
  8. Claude E Shannon and Warren Weaver. 1949. A mathematical model of com- munication. Urbana, IL: University of Illinois Press 11 (1949), 11-20.
  9. Ben Shneiderman and Catherine Plaisant. 2010. Designing the user interface: Strategies for efective human-computer interaction. Pearson Education India.
  10. David R Thomas. 2006. A general inductive approach for analyzing qualitative evaluation data. American journal of evaluation 27, 2 (2006), 237-246.
  11. Xuhai Xu, Han Zhang, Yasaman Sefdgar, Yiyi Ren, Xin Liu, Woosuk Seo, Jennifer Brown, Kevin Kuehn, Mike Merrill, Paula Nurius, Shwetak Patel, Tim Althof, Margaret Morris, Eve Riskin, Jennifer Mankof, and Anind Dey. 2022. GLOBEM Dataset: Multi-Year Datasets for Longitudinal Human Behavior Modeling Gen- eralization. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 24655-24692.
  12. Kun-Hsing Yu, Andrew L Beam, and Isaac S Kohane. 2018. Artifcial intelligence in healthcare. Nature biomedical engineering 2, 10 (2018), 719-731.

Cited by

Sorted by citations

Authors
Zhang, S., Yu, J., Xu, X., Yin, C., Lu, Y., Yao, B., Tory, M., Padilla, L. M., Caterino, J., Zhang, P., & Wang, D.
Affiliations
  • Northeastern University (US)
  • University of Hong Kong (HK)
  • Massachusetts Institute of Technology (US)
  • The Ohio State University (US)
  • Rensselaer Polytechnic Institute (US)
Corresponding author
  • Zhang, Ping
  • Wang, Dakuo
Venue
Proceedings of the CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2024-05-11
Volume
2024
Pages
1-18
OpenAlex
W4396832329
Cited by
99
References
96
Citations by year
2026: 43 · 2025: 47 · 2024: 8 · 2023: 1
FWCI
26.35
Type
conference paper
Language
en
Open access
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (IIS-2145625)
  • National Institutes of Health (R01GM141279)
  • NIH (National Institutes of Health) (1R01MD018424-01)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
human-AI collaboration, sepsis diagnosis, clinical decision-making, clinical decision support, early diagnosis, electronic health records, heuristic evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review