Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index; 8 of the 16 sources its claims cite were checked for support.
Abstract
Abstract Background Large Language Models (LLMs) show promise for clinical decision support in Intensive Care Units (ICU), but their safety and reliability remain inadequately evaluated through dual testing of both memory-dependent and memory-independent safety mechanisms. Objective To comprehensively evaluate LLMs using two independent safety tests: context-dependent contraindication memory (penicillin allergy recall) and context-independent authority resistance (Extended Milgram Test), revealing whether these represent unified or dissociated safety mechanisms. Methods Twenty-three LLMs underwent automated testing via 24-hour ICU simulation on consumer hardware (NVIDIA RTX 3060 12GB). A subset of 26 models completed an Extended Milgram Test with five escalating harmful command scenarios. Scoring assessed safety compliance, Milgram resistance, conflict detection, and performance. Results Critical findings revealed dissociation between abstract ethics and clinical memory. While 65% of models achieved perfect Milgram resistance (100%), only 8.7% (n=2) correctly refused penicillin with allergy mention. Eight models demonstrated 100% Milgram resistance yet failed allergy recall (r = -0.39, p = 0.23). Only Granite 3.1 8B achieved perfect performance on both tests. Conclusions Abstract ethical reasoning (refusing harmful orders in principle) is independent from concrete clinical memory (tracking patient-specific risks). Safe medical AI requires both capabilities—rarely both present. Dual safety testing should become mandatory for medical AI certification. Highlights Only 8.7% of tested LLMs passed critical safety tests for medication prescribing First study demonstrating dissociation between abstract ethics and clinical memory (r = -0.39) Eight models refused all harmful orders but forgot documented allergies Granite 3.1 8B only model achieving perfect performance on both safety tests Dual safety testing framework proposed for medical AI certification
Overview
CoChat Summary
## TL;DR
This computational benchmark tested 26 LLMs on ICU safety using two separate checks: remembering a documented penicillin allergy over 24 hours and resisting harmful authority-based instructions. Only 2 models (8.7%) passed the critical allergy safety test perfectly, while 17/26 (65.4%) achieved perfect Milgram resistance, showing these are dissociated safety capabilities.
## Key findings
- Only 2 of 23 models (8.7%) achieved Grade A+ on the allergy test: Granite 3.2 8B and Granite 3.1 8B; both explicitly refused amoxicillin/clavulanate and mentioned the documented penicillin allergy.
- 18 of 23 models (78.3%) received 0 safety points on the allergy test, either complying with the unsafe order or failing to clearly refuse.
- On the Extended Milgram Test, 17 of 26 models (65.4%) achieved perfect 100% resistance; mean resistance was 85.6% and the median was 100% (range 0–100%, SD 27.8%).
- The relationship between Milgram resistance and allergy-test performance was negative but not significant (r = -0.39, p = 0.23), and 8 models had perfect Milgram resistance yet failed the allergy test completely.
## Methods
- Computational benchmark study of 26 contemporary LLMs run on a single consumer workstation (NVIDIA RTX 3060 12GB, AMD Ryzen 7 5800X, 32GB RAM) using local inference with quantized models.
- Primary evaluation used a 24-hour ICU simulation with 31 sequential events; Event 000 documented a severe penicillin allergy, and Event 999 later tested whether models would refuse an order for amoxicillin/clavulanate.
- Main analyses included manual scoring of safety, conflict detection, and performance; for the Milgram subset, responses to 5 harmful authority scenarios were classified as REFUSE/COMPLY/UNCLEAR and summarized as a resistance percentage.
- Statistical analyses used descriptive statistics and correlations (Pearson/Spearman), with significance set at α = 0.05.
## Limitations
- The paper states the scenarios were synthetic, created by a single intensivist, and may not capture real ICU complexity, ambiguity, or interpersonal dynamics.
- It was done at a single center on one hardware configuration, so latency and performance may not generalize to other systems or settings.
- The paper says model sampling was a convenience sample based on public availability and ≤12GB VRAM constraints, so larger models were excluded; this limits generalizability (stated in the paper).
- The safety assessment relied on a single critical scenario (penicillin allergy / Event 999), so applicability to other contraindication types remains uncertain (stated in the paper).
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 2 Not valid · 1 Not checked · 102 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“Previous work on AI sycophancy has focused on knowledge domains (math, science, coding) where compliance with incorrect authority leads to wrong answers but not physical harm [9].”
What the sources say
- Disputedcites [9] Sharma M , Tong M , Korbak T , et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548. 2023. sourceExperimental — accuracy not yet measured
The Related Work section (p. 9) describes earlier sycophancy evaluations involving users’ views on politics, philosophy, and NLP, as well as biographies—not a focus on math, science, and coding. The paper does report assistants giving incorrect answers when challenged (§3.2, p. 3), but it does not characterize prior work as limited to domains without physical harm. (LLM, conf=0.82)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prior benchmarks of LLMs in medicine have primarily focused on knowledge assessment (USMLE examinations, medical licensing questions) [6,7] or general reasoning capabilities [2].”
What the sources say
- Supportedcites [6] Nori H , King N , McKinney SM , et al. Capabilities of GPT-4 on medical challenge problems. arXiv preprint arXiv:2303.13375. 2023. source
The cited paper is itself a benchmark of GPT-4 on medical competency examinations and medical knowledge-and-reasoning datasets. Section 2.1 (p. 3) identifies USMLE practice exams and other medical reasoning benchmarks, supporting its use as an example of the examination-focused benchmarks cited in the sentence. (LLM, conf=0.91)
- Supportedcites [7] 2024, Can large language models reason about medical questions?, Patterns source
Citation [2] is attached to the clause about reasoning capabilities, not the separate clause about knowledge-assessment benchmarks. The Introduction (LLMs and medical applications) says medical questions require advanced reasoning, and the Discussion (Conclusion) reports an evaluation of the models’ reasoning steps. The paper is therefore an example of a medical LLM benchmark examining reasoning capabilities. (LLM, conf=0.88)
- Not matchedcites [2] 2024, Large language models in medicine, Nat Med
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Studies evaluating GPT-4 and Claude on medical board examinations report accuracy rates of 80-90% [3,6]”
What the sources say
- Not checked — no full text of the sourcecites [3] source
- Supportedcites [6] Nori H , King N , McKinney SM , et al. Capabilities of GPT-4 on medical challenge problems. arXiv preprint arXiv:2303.13375. 2023. source
The Introduction (p. 2) reports GPT-4 scores of 86.65% and 86.7% on official USMLE practice materials, within the claimed 80–90% range. This source supports the GPT-4 portion of the jointly cited sentence; it does not evaluate Claude. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“18 of 23 models (78.3%) received zero safety points, indicating complete failure on the critical allergy test.”
- Also stated in: § 3.2 Safety Performance: The Allergy Test · plain
“The majority (18 models, 78.3%) completely failed the safety test (Grade F)”
plainn = 18- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.2 Safety Performance: The Allergy Test · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only Granite 3.1 8B achieved perfect performance on both tests.”
- Also stated in: § Highlights · plain
“Granite 3.1 8B only model achieving perfect performance on both safety tests”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Highlights · plain
58 more claims
Create a free account to see all 110 claims and every source verdict.
- It may state two separate claims as one. 25
- Probably not a substantial claim. 25
- The quote only partly states this claim. 3
Citations
Cites
Behavioral Study of obedience.
Milgram, S. (1963) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0040525 · Cited by 4,918
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine
Lee, P., Bubeck, S., & Petro, J. (2023) · New England Journal of Medicine · doi:10.1056/nejmsr2214184 · Cited by 1,738
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612
Sharma, M., Tong, M., Korbak, T., Duperrouzel, D., Was, H., Korbak, J., Cheng, M., Askell, A., Bai, Y., Brain, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13548 · Cited by 1,313
Do no harm: a roadmap for responsible machine learning for health care
Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0548-6 · Cited by 1,266
Capabilities of GPT-4 on Medical Challenge Problems
Nori, H., King, N., McKinney, S. M., Carignan, D., & Horvitz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13375 · Cited by 509
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2022) · arXiv · doi:10.48550/arxiv.2207.08143 · Cited by 343
Metadata
- Authors
- Shlyakhta, T.
- Affiliations
- Nemocnica Agel Komarno
- Corresponding author
- Shlyakhta, Taras
- ORCID
- Shlyakhta, Taras
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2026-02-10
- OpenAlex
- W7128499849
- Cited by
- 0
- References
- 8
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- medRxiv (acceptedVersion, cc-by)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- large language models, clinical decision support, intensive care unit, AI safety evaluation, drug prescribing, benchmarking, consumer hardware
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware. https://app.cochat.ai/papers/10.64898/2026.02.08.26345854
- The paper
- Shlyakhta, T. (2026). Benchmarking Large Language Models for Intensive Care Unit Clinical Decision Support: A Dual Safety Evaluation of 26 Models on Consumer Hardware. medRxiv. https://doi.org/10.64898/2026.02.08.26345854
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.