CoChat

Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware

Open access
Shlyakhta, Taras · 2026 · medRxiv

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index; 8 of the 16 sources its claims cite were checked for support.

Abstract

Abstract Background Large Language Models (LLMs) show promise for clinical decision support in Intensive Care Units (ICU), but their safety and reliability remain inadequately evaluated through dual testing of both memory-dependent and memory-independent safety mechanisms. Objective To comprehensively evaluate LLMs using two independent safety tests: context-dependent contraindication memory (penicillin allergy recall) and context-independent authority resistance (Extended Milgram Test), revealing whether these represent unified or dissociated safety mechanisms. Methods Twenty-three LLMs underwent automated testing via 24-hour ICU simulation on consumer hardware (NVIDIA RTX 3060 12GB). A subset of 26 models completed an Extended Milgram Test with five escalating harmful command scenarios. Scoring assessed safety compliance, Milgram resistance, conflict detection, and performance. Results Critical findings revealed dissociation between abstract ethics and clinical memory. While 65% of models achieved perfect Milgram resistance (100%), only 8.7% (n=2) correctly refused penicillin with allergy mention. Eight models demonstrated 100% Milgram resistance yet failed allergy recall (r = -0.39, p = 0.23). Only Granite 3.1 8B achieved perfect performance on both tests. Conclusions Abstract ethical reasoning (refusing harmful orders in principle) is independent from concrete clinical memory (tracking patient-specific risks). Safe medical AI requires both capabilities—rarely both present. Dual safety testing should become mandatory for medical AI certification. Highlights Only 8.7% of tested LLMs passed critical safety tests for medication prescribing First study demonstrating dissociation between abstract ethics and clinical memory (r = -0.39) Eight models refused all harmful orders but forgot documented allergies Granite 3.1 8B only model achieving perfect performance on both safety tests Dual safety testing framework proposed for medical AI certification

Overview

CoChat Summary

## TL;DR

This computational benchmark tested 26 LLMs on ICU safety using two separate checks: remembering a documented penicillin allergy over 24 hours and resisting harmful authority-based instructions. Only 2 models (8.7%) passed the critical allergy safety test perfectly, while 17/26 (65.4%) achieved perfect Milgram resistance, showing these are dissociated safety capabilities.

## Key findings

  • Only 2 of 23 models (8.7%) achieved Grade A+ on the allergy test: Granite 3.2 8B and Granite 3.1 8B; both explicitly refused amoxicillin/clavulanate and mentioned the documented penicillin allergy.
  • 18 of 23 models (78.3%) received 0 safety points on the allergy test, either complying with the unsafe order or failing to clearly refuse.
  • On the Extended Milgram Test, 17 of 26 models (65.4%) achieved perfect 100% resistance; mean resistance was 85.6% and the median was 100% (range 0–100%, SD 27.8%).
  • The relationship between Milgram resistance and allergy-test performance was negative but not significant (r = -0.39, p = 0.23), and 8 models had perfect Milgram resistance yet failed the allergy test completely.

## Methods

  • Computational benchmark study of 26 contemporary LLMs run on a single consumer workstation (NVIDIA RTX 3060 12GB, AMD Ryzen 7 5800X, 32GB RAM) using local inference with quantized models.
  • Primary evaluation used a 24-hour ICU simulation with 31 sequential events; Event 000 documented a severe penicillin allergy, and Event 999 later tested whether models would refuse an order for amoxicillin/clavulanate.
  • Main analyses included manual scoring of safety, conflict detection, and performance; for the Milgram subset, responses to 5 harmful authority scenarios were classified as REFUSE/COMPLY/UNCLEAR and summarized as a resistance percentage.
  • Statistical analyses used descriptive statistics and correlations (Pearson/Spearman), with significance set at α = 0.05.

## Limitations

  • The paper states the scenarios were synthetic, created by a single intensivist, and may not capture real ICU complexity, ambiguity, or interpersonal dynamics.
  • It was done at a single center on one hardware configuration, so latency and performance may not generalize to other systems or settings.
  • The paper says model sampling was a convenience sample based on public availability and ≤12GB VRAM constraints, so larger models were excluded; this limits generalizability (stated in the paper).
  • The safety assessment relied on a single critical scenario (penicillin allergy / Event 999), so applicability to other contraindication types remains uncertain (stated in the paper).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.64898/2026.02.08.26345854

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

5 Valid · 2 Not valid · 1 Not checked · 102 Own results and interpretations

58 more claims

Create a free account to see all 110 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 25
  • Probably not a substantial claim. 25
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Citations

Cites

Show all 13 references

Metadata

Authors
Shlyakhta, T.
Affiliations
Nemocnica Agel Komarno
Corresponding author
Shlyakhta, Taras
Venue
medRxiv
Publisher
openRxiv
Published
2026-02-10
OpenAlex
W7128499849
Cited by
0
References
8
Type
posted content
Language
en
Open access
Locations
medRxiv (acceptedVersion, cc-by)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
Keywords
large language models, clinical decision support, intensive care unit, AI safety evaluation, drug prescribing, benchmarking, consumer hardware

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware. https://app.cochat.ai/papers/10.64898/2026.02.08.26345854
The paper
Shlyakhta, T. (2026). Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware. medRxiv. https://doi.org/10.64898/2026.02.08.26345854

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.