CoChat

Towards Trustworthy AI in Healthcare: Epistemic Uncertainty Estimation for Clinical Decision Support

Open access

The DOI matches its Crossref record; 48 of the 49 references it cites were found in a registry or an index, and 1 are not checked yet.

Abstract

Introduction: Widespread adoption of AI for medical decision-making is still hindered due to ethical and safety-related concerns. For AI-based decision support systems in healthcare settings, it is paramount to be reliable and trustworthy. Common deep learning approaches, however, have the tendency towards overconfidence when faced with unfamiliar or changing conditions. Inappropriate extrapolation beyond well-supported scenarios may have dire consequences highlighting the importance of the reliable estimation of local knowledge uncertainty and its communication to the end user. Materials and Methods: While neural network ensembles (ENNs) have been heralded as a potential solution to these issues for many years, deep learning methods, specifically modeling the amount of knowledge, promise more principled and reliable behavior. This study compares their reliability in clinical applications. We centered our analysis on experiments with low-dimensional toy datasets and the exemplary case study of mortality prediction for intensive care unit hospitalizations using Electronic Health Records (EHRs) from the MIMIC3 study. For predictions on the EHR time series, Encoder-Only Transformer models were employed. Knowledge uncertainty estimation is achieved with both ensemble and Spectral Normalized Neural Gaussian Process (SNGP) variants of the common Transformer model. We designed two datasets to test their reliability in detecting token level and more subtle discrepancies both for toy datasets and an EHR dataset. Results: While both SNGP and ENN model variants achieve similar prediction performance (AUROC: ≈0.85, AUPRC: ≈0.52 for in-hospital mortality prediction from a selected MIMIC3 benchmark), the former demonstrates improved capabilities to quantify knowledge uncertainty for individual samples/patients. Discussion/Conclusions: Methods including a knowledge model, such as SNGP, offer superior uncertainty estimation compared to traditional stochastic deep learning, leading to more trustworthy and safe clinical decision support.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/jpm15020058

Cites

48 of 49 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure A1 Visual representation of ENN and SNGP general behavior. Top: Exemplary problem. Bottom: Behavior along the cut plane s. The ENN method shows typical diverging behavior resulting in a measurable signal for knowledge uncertainty. The SNGP method shows approximated Gaussian Process behavior with the variance (represented by the orange 95% confidence interval) directly usable as signal for knowledge uncertainty.
Figure A1 Visual representation of ENN and SNGP general behavior. Top: Exemplary problem. Bottom: Behavior along the cut plane s. The ENN method shows typical diverging behavior resulting in a measurable signal for knowledge uncertainty. The SNGP method shows approximated Gaussian Process behavior with the variance (represented by the orange 95% confidence interval) directly usable as signal for knowledge uncertainty.CC BY · PubMed Central
Figure 1 Left: The common two moons dataset and the stripes dataset. Right: Synthetic OoD datasets for both the two moons and the stripes dataset. The colorbar indicates the distance to the nearest known data point from the source dataset.
Figure 1 Left: The common two moons dataset and the stripes dataset. Right: Synthetic OoD datasets for both the two moons and the stripes dataset. The colorbar indicates the distance to the nearest known data point from the source dataset.CC BY · PubMed Central
Figure 2 Results from the toy data experiments. Top: Class predictions. Bottom: KU measures. Left: ENN. Right: SNGP.
Figure 2 Results from the toy data experiments. Top: Class predictions. Bottom: KU measures. Left: ENN. Right: SNGP.CC BY · PubMed Central
Figure 3 OoD detection experiments on the two moons and stripes datasets. Top: KU heat maps with red markers indicating samples from the OoD datasets that are not recognized as OoD by the threshold. Bottom: EU values of the test dataset are shown as a kernel density estimate (ptest(log(10σ)) in orange together with the threshold (black line) based on the 90th quantile. KU values of the OoD samples are shown as blue dots. Left: ENN. Right: SNGP.
Figure 3 OoD detection experiments on the two moons and stripes datasets. Top: KU heat maps with red markers indicating samples from the OoD datasets that are not recognized as OoD by the threshold. Bottom: EU values of the test dataset are shown as a kernel density estimate (ptest(log(10σ)) in orange together with the threshold (black line) based on the 90th quantile. KU values of the OoD samples are shown as blue dots. Left: ENN. Right: SNGP.CC BY · PubMed Central
Figure 4 OoD detection experiments PTS and RTR on the MIMIC3 benchmark dataset. Left: ENN. Right: SNGP. Uncertainty measures of the test dataset are shown as the kernel density estimation (ptest(log(10σ+1)) in orange together with the threshold (black line) based on the 70th quantile. KU values of the OoD samples are shown as points. Points are colored according to the number of tokens for each sample.
Figure 4 OoD detection experiments PTS and RTR on the MIMIC3 benchmark dataset. Left: ENN. Right: SNGP. Uncertainty measures of the test dataset are shown as the kernel density estimation (ptest(log(10σ+1)) in orange together with the threshold (black line) based on the 70th quantile. KU values of the OoD samples are shown as points. Points are colored according to the number of tokens for each sample.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 49 references

Research works printed but not linked: 6

  1. Gal, Y. (2016). Uncertainty in Deep Learning. [Ph.D. Thesis, University of Cambridge].
  2. Henry, 2016, Adoption of Electronic Health Record Systems among US Non-Federal Acute Care Hospitals: 2008–2015, ONC Data Brief, 35, 2008
  3. Qiu, R. (2020). Modeling Uncertainty in Deep Learning Models of Electronic Health Records. [Ph.D. Thesis, University of North Carolina at Charlotte].
  4. Qiu, R., Jia, Y., Hadzikadic, M., Dulin, M., Niu, X., and Wang, X. (2019). Modeling the Uncertainty in Electronic Health Records: A Bayesian Deep Learning Approach. arXiv.
  5. Huang, Y., Zhang, H., Shi, Y., Kolter, J.Z., and Anandkumar, A. (2021). Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds. arXiv.
  6. Qi, X., Wang, J., Chen, Y., Shi, Y., and Zhang, L. (2023). LipsFormer: Introducing Lipschitz Continuity to Vision Transformers. arXiv.

Cited by

See all 17

Metadata

Authors
Lindenmeyer, A., Blattmann, M., Franke, S., Neumuth, T., & Schneider, D.
Affiliations
  • Center for Scalable Data Analytics and Artificial Intelligence (DE)
  • Leipzig University (DE)
Corresponding author
Lindenmeyer, Adrian
Venue
Journal of Personalized Medicine
Publisher
MDPI AG
Published
2025-01-31
Volume
15
Issue
2
Pages
58
ISSN
2075-4426 (ISSN-L)
OpenAlex
W4407039475
Cited by
17
References
43
Citations by year
2026: 14 · 2025: 3
FWCI
6.79
Type
journal article
Language
en
Open access
Locations
  • Journal of Personalized Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
Federal Ministry of Education and Research of Germany (ScaDS.AI)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
trustworthy AI, clinical decision support, epistemic uncertainty, neural network ensembles, encoder-only transformer, in-hospital mortality prediction, intensive care unit, electronic health records, MIMIC-III, overconfidence, uncertainty quantification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Towards Trustworthy AI in Healthcare: Epistemic Uncertainty Estimation for Clinical Decision Support. https://app.cochat.ai/papers/10.3390/jpm15020058
The paper
Lindenmeyer, A., Blattmann, M., Franke, S., Neumuth, T., & Schneider, D. (2025). Towards Trustworthy AI in Healthcare: Epistemic Uncertainty Estimation for Clinical Decision Support. Journal of Personalized Medicine. https://doi.org/10.3390/jpm15020058

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.