CoChat
Sign Up
PreprintCited by 2Machine Learning in HealthcareArtificial Intelligence in Healthcare and EducationAdversarial Robustness in Machine Learning

The DOI matches its DataCite record.

Inadequacy of common stochastic neural networks for reliable clinical decision support

Open access
Lindenmeyer, Adrian, Blattmann, Malte, Franke, Stefan, Neumuth, Thomas, Schneider, Daniel·2024·arXiv
DOI 10.48550/arxiv.2401.13657arXiv 2401.13657
Open this paper in CoChat — free accountOpen PDF

Abstract

Widespread adoption of AI for medical decision making is still hindered due to ethical and safety-related concerns. For AI-based decision support systems in healthcare settings it is paramount to be reliable and trustworthy. Common deep learning approaches, however, have the tendency towards overconfidence under data shift. Such inappropriate extrapolation beyond evidence-based scenarios may have dire consequences. This highlights the importance of reliable estimation of local uncertainty and its communication to the end user. While stochastic neural networks have been heralded as a potential solution to these issues, this study investigates their actual reliability in clinical applications. We centered our analysis on the exemplary use case of mortality prediction for ICU hospitalizations using EHR from MIMIC3 study. For predictions on the EHR time series, Encoder-Only Transformer models were employed. Stochasticity of model functions was achieved by incorporating common methods such as Bayesian neural network layers and model ensembles. Our models achieve state of the art performance in terms of discrimination performance (AUC ROC: 0.868+-0.011, AUC PR: 0.554+-0.034) and calibration on the mortality prediction benchmark. However, epistemic uncertainty is critically underestimated by the selected stochastic deep learning methods. A heuristic proof for the responsible collapse of the posterior distribution is provided. Our findings reveal the inadequacy of commonly used stochastic deep learning approaches to reliably recognize OoD samples. In both methods, unsubstantiated model confidence is not prevented due to strongly biased functional posteriors, rendering them inappropriate for reliable clinical decision support. This highlights the need for approaches with more strictly enforced or inherent distance-awareness to known data points, e.g., using kernel-based techniques.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2401.13657

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 56

  1. M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, Li Liu, M. Ghavamzadeh, P. Fieguth, Xi- aochun Cao, A. Khosravi, U. R. Acharya, V. Makarenkov, and S. Nahavandi. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76: 243-297, 12 2021. doi:10.1016/j.inffus.2021.05.008. · DOI · Paper page
  2. E. Begoli, T. Bhattacharya, and D. Kusnezov. The need for uncertainty quantification in machine- assisted medical decision making. Nature Machine Intelligence, 1:20-23, 1 2019. doi:10.1038/ s42256-018-0004-1. · DOI · Paper page
  3. P. Blinov, M. Avetisian, V. Kokh, D. Umerenkov, and A. Tuzhilin. Predicting clinical diagnosis from patients electronic health records using bert-based neural networks. In Artificial Intelligence in Medicine: 18th International Conference on Artificial Intelligence in Medicine, pages 111-121, 2020. doi:10.1007/978-3-030-59137-3-11. · DOI
  4. C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural networks. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 1613- 1622, 07 2015.
  5. Edward Choi, M. T. Bahadori, J. A. Kulas, A. Schuetz, W. F. Stewart, and Jinmeng Sun. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Proceedings of the 30th International Conference on Neural Information Processing Systems, page 3512-3520, 12 2016.
  6. Edward Choi, M. T. Bahadori, A. Schuetz, W. F. Stewart, and Jimeng Sun. Doctor ai: Predict- ing clinical events via recurrent neural networks. In Proceedings of the 1st Machine Learning for Healthcare Conference, volume 56, pages 301-318, 08 2016.
  7. F. D'Angelo, V. Fortuin, and F. Wenzel. On stein variational neural network ensembles. 2021. arXiv:2106.10760 [cs.LG]. · arXiv
  8. S. Darabi, M. Kachuee, S. Fazeli, and M. Sarrafzadeh. Taper: Time-aware patient ehr representation. IEEE Journal of Biomedical and Health Informatics, 24(11):3268-3275, 11 2020. doi:10.1109/ jbhi.2020.2984931. · DOI
  9. J. Devlin, Ming-Wei Chang, Kenton Lee, and K. Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume 1, pages 4171-4186, 06 2019. doi:10.18653/v1/n19-1423. · DOI · Paper page
  10. M. W. Dusenberry, Dustin Tran, Edward Choi, J. Kemp, J. Nixon, G. Jerfel, K. Heller, and Andrew M. Dai. Analyzing the role of model uncertainty for electronic health records. In Pro- ceedings of the ACM Conference on Health, Inference, and Learning, pages 204-213, 04 2020. doi:10.1145/3368555.3384457. · DOI
  11. V. Dwaracherla, Xiuyuan Lu, M. Ibrahimi, I. Osband, Zheng Wen, and B. Van Roy. Hypermodels for exploration. 2020. arXiv:2006.07464 [cs.LG]. · arXiv
  12. S. Farquhar, M. A. Osborne, and Y. Gal. Radial bayesian neural networks: Beyond discrete support in large-scale bayesian deep learning. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108, pages 1352-1362, 08 2020.
  13. Andrew Foong, D. Burt, Yingzhen Li, and R. Turner. On the expressiveness of approximate inference in bayesian neural networks. In Advances in Neural Information Processing Systems, volume 33, pages 15897-15908, 2020.
  14. Y. Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
  15. A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, Chung-Kang Peng, and H. E. Stanley. Physiobank, physiotoolkit, and physionet: Components of a new research resource for complex physiologic signals. Circulation, 101(23):e215- e220, 2000. doi:10.1161/01.cir.101.23.e215. · DOI · Paper page
  16. H. Harutyunyan, H. Khachatrian, D. C. Kale, G. V. Steeg, and A. Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific Data, 6(96), 6 2019. doi:10.1038/ s41597-019-0103-9. · DOI · Paper page
  17. A. E. W. Johnson, T. J. Pollard, and R. G. Mark. Mimic-iii clinical database (version 1.4). 2016. doi:10.13026/c2xw26. · DOI · Paper page
  18. A. E. W. Johnson, T. J. Pollard, Lu Shen, Li-Wei H. Lehman, Mengling Feng, M. Ghassemi, B. Moody, P. Szolovits, L. A. Celi, and R. G. Mark. Mimic-iii, a freely accessible critical care database. Scientific Data, 3(160035), 05 2016. doi:10.1038/sdata.2016.35. · DOI · Paper page
  19. L. V. Jospin, H. Laga, F. Boussaid, W. Buntine, and M. Bennamoun. Hands-on bayesian neural networks-a tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2): 29-48, 2022. doi:10.1109/mci.2022.3155327. · DOI · Paper page
  20. D. P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2017. arXiv:1412.6980 [cs.LG]. · arXiv
  21. I. D. Kivlichan, Zi Lin, Jeremiah Liu, and L. Vasserman. Measuring and improving model-moderator collaboration using uncertainty estimation. 2021. arXiv:2107.04212 [cs.LG]. · arXiv
  22. B. Kompa, J. Snoek, and A. L. Beam. Second opinion needed: Communicating uncertainty in med- ical machine learning. NPJ Digital Medicine, 4(1):4, 1 2021. doi:10.1038/s41746-020-00367-3. · DOI · Paper page
  23. M. Kull, M. P. Nieto, M. Kängsepp, T. S. Filho, Hao Song, and P. Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In Advances in Neural Information Processing Systems, volume 32, 2019.
  24. B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, 2017.
  25. R. S. Ledley and L. B. Lusted. Reasoning foundations of medical diagnosis: Symbolic logic, prob- ability, and value theory aid our understanding of how physicians reason. Science, 130(3366):9-21, 07 1959. doi:10.1126/science.130.3366.9. · DOI
  26. Yikuan Li, Shishir Rao, J. R. A. Solares, A. Hassaine, R. Ramakrishnan, D. Canoy, Yajie Zhu, K. Rahimi, and G. Salimi-Khorshidi. Behrt: Transformer for electronic health records. Scientific Reports, 10(7155), 04 2020. doi:10.1038/s41598-020-62922-y. · DOI · Paper page
  27. Yikuan Li, Shishir Rao, A. Hassaine, Ramakrishnan R., D. Canoy, G. Salimi-Khorshidi, M. Mamouei, T. Lukasiewicz, and K. Rahimi. Deep bayesian gaussian processes for uncer- tainty estimation in electronic health records. Scientific reports, 11(20685), 10 2021. doi: 10.1038/s41598-021-00144-6. · DOI
  28. Jun Liang, Ying Li, Zhang Zhang, Dongxia Shen, Jie Xu, Xu Zheng, Tong Wang, Buzhou Tang, Jianbo Lei, and Jiajie Zhang. Adoption of electronic health records (ehrs) in china during the past 10 years: Consecutive survey data analysis and comparison of sino-american challenges and experiences. Journal of Medical Internet Research, 23(2):e24813, 02 2021. doi:10.2196/24813. · DOI
  29. Zhan Wei Lim, Mong Li Lee, Wynne Hsu, and Tien Yin Wong. Building trust in deep learning system towards automated disease detection. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):9516-9521, 07 2019. doi:10.1609/aaai.v33i01.33019516. · DOI
  30. Jeremiah Liu, Zi Lin, S. Padhy, Dustin Tran, T. Bedrax Weiss, and B. Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. In Advances in Neural Information Processing Systems, volume 33, pages 7498-7512, 2020.
  31. S. J. MacEachern and N. D. Forkert. Machine learning for precision medicine. Genome, 64(4): 416-425, 2021. doi:10.1139/gen-2020-0131. · DOI
  32. D. J. C. MacKay. A Practical Bayesian Framework for Backpropagation Networks. Neural Compu- tation, 4(3):448-472, 05 1992. doi:10.1162/neco.1992.4.3.448. · DOI · Paper page
  33. N. Menachemi and T. H. Collum. Benefits and drawbacks of electronic health record systems. Risk Management and Healthcare Policy, 4:47-55, 2011. doi:10.2147/rmhp.s12985. · DOI · Paper page
  34. J. Mullenbach, S. Wiegreffe, J. Duke, Jimeng Sun, and J. Eisenstein. Explainable prediction of medical codes from clinical text. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume 1, pages 1101-1111, 06 2018. doi:10.18653/v1/n18-1100. · DOI · Paper page
  35. J. Nazarovs, R. R. Mehta, V. S. Lokhande, and V. Singh. Graph reparameterizations for enabling 1000+ monte carlo iterations in bayesian deep neural networks. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161, pages 118-128, 07 2021.
  36. R. M. Neal. Bayesian Learning For Neural Networks. Springer New York, 1996. doi:10.1007/ 978-1-4612-0745-0. · DOI · Paper page
  37. Chao Pang, Xinzhuo Jiang, K. S. Kalluri, M. Spotnitz, RuiJun Chen, A. Perotte, and K. Natarajan. Cehr-bert: Incorporating temporal information from structured ehr data to improve prediction tasks. In Proceedings of Machine Learning for Health, volume 158, pages 239-260, 12 2021.
  38. T. Pearce, F. Leibfried, and A. Brintrup. Uncertainty in neural networks: Approximately bayesian ensembling. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108, pages 234-244, 08 2020.
  39. Xueping Peng, Guodong Long, Tao Shen, Sen Wang, Jing Jiang, and M. Blumenstein. Temporal self-attention network for medical concept embedding. In 2019 IEEE International Conference on Data Mining, pages 498-507, 11 2019. doi:10.1109/icdm.2019.00060. · DOI
  40. Riyi Qiu. Modeling Uncertainty in Deep Learning Models of Electronic Health Records. PhD thesis, The University of North Carolina at Charlotte, 2020.
  41. Riyi Qiu, Yugang Jia, M. Hadzikadic, M. Dulin, Xi Niu, and Xin Wang. Modeling the uncertainty in electronic health records: A bayesian deep learning approach. 2019. arXiv:1907.06162 [cs.LG]. · arXiv
  42. W. Raghupathi and V. Raghupathi. Big data analytics in healthcare: Promise and potential. Health Information Science and Systems, 2(3), 2014. doi:10.1186/2047-2501-2-3. · DOI · Paper page
  43. L. Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-bert: Pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ Digital Medicine, 4(86), 05 2021. doi:10.1038/s41746-021-00455-y. · DOI · Paper page
  44. B. Ristevski and Ming Chen. Big data analytics in medicine and healthcare. Journal of Integrative Bioinformatics, 15(3):20170030, 2018. doi:10.1515/jib-2017-0030. · DOI
  45. S. Ruder. An overview of gradient descent optimization algorithms. 2017. arXiv:1609.04747 [cs.LG]. · arXiv
  46. Junyuan Shang, Tangfei Ma, Cao Xiao, and Jimeng Sun. Pre-training of graph augmented trans- formers for medication recommendation. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pages 5953-5959, 07 2019. doi:10.24963/ijcai.2019/825. · DOI · Paper page
  47. B. Shickel, P.J. Tighe, A. Bihorac, and P. Rashidi. Deep ehr: A survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE Journal of Biomedical and Health Informatics, 22(5):1589-1604, 09 2018. doi:10.1109/jbhi.2017.2767063. · DOI · Paper page
  48. Yuqi Si, Jingcheng Du, Zhao Li, Xiaoqian Jiang, T. Miller, Fei Wang, W. Jim Zheng, and K. Roberts. Deep representation learning of patient data from electronic health records (ehr): A systematic re- view. Journal of Biomedical Informatics, 115(103671), 03 2021. doi:10.1016/j.jbi.2020.103671. · DOI · Paper page
  49. R. C. Smith. Uncertainty Quantification: Theory, Implementation, and Applications. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2013. doi:10.1137/1.9781611973228. · DOI
  50. Huan Song, D. Rajan, J. J. Thiagarajan, and A. Spanias. Attend and diagnose: Clinical time series analysis using attention models. 2017. arXiv:1711.03905 [stat.ML]. · arXiv
  51. T. J. Sullivan. Measures of Information and Uncertainty, pages 75-90. Springer International Publishing, Cham, 2015. doi:10.1007/978-3-319-23395-6-5. · DOI
  52. S. Tonekaboni, S. Joshi, M. D. McCradden, and A. Goldenberg. What clinicians want: Contextual- izing explainable machine learning for clinical end use. In Proceedings of the 4th Machine Learning for Healthcare Conference, volume 106, pages 359-380, 08 2019.
  53. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
  54. Ying Wang, Xiao Xu, Tao Jin, Xiang Li, Guotong Xie, and Jianmin Wang. Inpatient2vec: Medical representation learning for inpatients. In 2019 IEEE International Conference on Bioinformatics and Biomedicine, pages 1113-1117, 11 2019. doi:10.1109/bibm47256.2019.8983281. · DOI
  55. F. Wenzel, J. Snoek, Dustin Tran, and R. Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. In Advances in Neural Information Processing Systems, volume 33, pages 6514-6527, 2020.
  56. Po-Yen Wu, Chih-Wen Cheng, C. D. Kaddi, J. Venugopalan, R. Hoffman, and May Dongmei Wang. -omic and electronic health record big data analytics for precision medicine. IEEE Transactions on Biomedical Engineering, 64(2):263-273, 2017. doi:10.1109/tbme.2016.2573285. · DOI

Cited by

Sorted by citations

See all 2

Metadata

Authors
Lindenmeyer, A., Blattmann, M., Franke, S., Neumuth, T., & Schneider, D.
Affiliations
  • Center for Scalable Data Analytics and Artificial Intelligence (DE)
  • Leipzig University (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2401.13657
arXiv
2401.13657
OpenAlex
W4391272199
Cited by
2
References
0
Citations by year
2025: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Deutsche Forschungsgemeinschaft (460234259)
  • Bundesministerium für Bildung und Forschung
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
stochastic neural networks, epistemic uncertainty, uncertainty calibration, out-of-distribution detection, clinical decision support, mortality prediction, intensive care unit patients, Bayesian neural networks, ensemble learning, encoder-only transformer, MIMIC-III, posterior collapse, overconfidence, electronic health records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Inadequacy of common stochastic neural networks for reliable clinical decision support. https://app.cochat.ai/papers/10.48550/arxiv.2401.13657
The paper
Lindenmeyer, A., Blattmann, M., Franke, S., Neumuth, T., & Schneider, D. (2024). Inadequacy of common stochastic neural networks for reliable clinical decision support. arXiv. https://doi.org/10.48550/arxiv.2401.13657