CoChat
Sign Up
Cited by 57Machine Learning in HealthcareArtificial Intelligence in Healthcare and EducationRadiomics and Machine Learning in Medical Imaging

Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data

Open access
Khader, Firas, Kather, Jakob Nikolas, Müller-Franzes, Gustav, Wang, Tianci, Han, Tianyu, Tayebi Arasteh, Soroosh, Hamesch, Karim, Bressem, Keno +5 more·2023·Scientific Reports
DOI 10.1038/s41598-023-37835-1PMC PMC10314902
Open this paper in CoChat — free accountOpen PDF

Abstract

When clinicians assess the prognosis of patients in intensive care, they take imaging and non-imaging data into account. In contrast, many traditional machine learning models rely on only one of these modalities, limiting their potential in medical applications. This work proposes and evaluates a transformer-based neural network as a novel AI architecture that integrates multimodal patient data, i.e., imaging data (chest radiographs) and non-imaging data (clinical data). We evaluate the performance of our model in a retrospective study with 6,125 patients in intensive care. We show that the combined model (area under the receiver operating characteristic curve [AUROC] of 0.863) is superior to the radiographs-only model (AUROC = 0.811, p < 0.001) and the clinical data-only model (AUROC = 0.785, p < 0.001) when tasked with predicting in-hospital survival per patient. Furthermore, we demonstrate that our proposed model is robust in cases where not all (clinical) data points are available.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-023-37835-1

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 34

  1. [1]Halpern NA Pastores SM Critical care medicine in the United States 2000–2005: An analysis of bed numbers, occupancy rates, payer mix, and costs Crit. Care Med. 2010 38 65 71 10.1097/CCM.0b013e3181b090d0 19730257 · DOI
  2. [2]Syed, M. et al. Application of machine learning in intensive care unit (ICU) settings using MIMIC dataset: systematic review. Informatics (MDPI)8, 16 (2021). PMC8112729 33981592
  3. [3]Pirracchio R Mortality prediction in intensive care units with the Super ICU Learner Algorithm (SICULA): A population-based study Lancet Respir. Med. 2015 3 42 52 10.1016/S2213-2600(14)70239-5 25466337 PMC4321691 · DOI · Paper page
  4. [4]Hoogendoorn, M., el Hassouni, A., Mok, K., Ghassemi, M. & Szolovits, P. Prediction using patient comparison vs. modeling: A case study for mortality prediction. in 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 2464–2467 (2016). 28268823
  5. [5]Awad A Bader-El-Den M McNicholas J Briggs J Early hospital mortality prediction of intensive care unit patients using an ensemble learning approach Int. J. Med. Informatics 2017 108 185 195 10.1016/j.ijmedinf.2017.10.002 29132626 · DOI · Paper page
  6. [6]Weissman GE Inclusion of unstructured clinical text improves early prediction of death or prolonged ICU stay Crit. Care Med. 2018 46 1125 1132 10.1097/CCM.0000000000003148 29629986 PMC6005735 · DOI
  7. [9]Bressem KK Comparing different deep learning architectures for classification of chest radiographs Sci. Rep. 2020 10 13590 10.1038/s41598-020-70479-z 32788602 PMC7423963 · DOI
  8. [10]Khader, F. et al. Artificial Intelligence for Clinical Interpretation of Bedside Chest Radiographs. Radiology 220510 (2022). 36472534
  9. [11]Spiritoso R Padley S Singh S Chest X-ray interpretation in UK intensive care units: A survey 2014 J. Intens. Care Soc. 2015 16 339 344 10.1177/1751143715580141 PMC5606453 28979441 · DOI
  10. [13]Vaswani, A. et al. Attention is all you need. in Advances in Neural Information Processing Systems vol. 30 5998–6008 (Curran Associates, Inc., 2017).
  11. [14]Hayat, N., Geras, K. J. & Shamout, F. E. Multi-modal fusion with clinical time-series data and chest X-ray images. Preprint at http://arxiv.org/abs/2207.07027 (2022). · arXiv
  12. [15]Hayat, N., Geras, K. J. & Shamout, F. E. Towards dynamic multi-modal phenotyping using chest radiographs and physiological data. http://arxiv.org/abs/2111.02710 (2021). · arXiv
  13. [16]Pölsterl, S., Wolf, T. N. & Wachinger, C. Combining 3D image and tabular data via the dynamic affine feature map transform. in Medical Image Computing and Computer Assisted Intervention—MICCAI 2021 688–698 (2021).
  14. [17]Sharma A Vans E Shigemizu D Boroevich KA Tsunoda T DeepInsight: A methodology to transform a non-image data to an image for convolution neural network architecture Sci. Rep. 2019 9 11399 10.1038/s41598-019-47765-6 31388036 PMC6684600 · DOI
  15. [18]Pascanu, R., Mikolov, T. & Bengio, Y. On the difficulty of training recurrent neural networks. in Proceedings of the 30th International Conference on Machine Learning 1310–1318 (PMLR, 2013).
  16. [20]Mettler FA Radiologic and nuclear medicine studies in the United States and worldwide: Frequency, radiation dose, and comparison with other radiation sources—1950–2007 Radiology 2009 253 520 531 10.1148/radiol.2532082010 19789227 · DOI
  17. [23]Song D Deep relation transformer for diagnosing glaucoma with optical coherence tomography and visual field function IEEE Trans. Med. Imaging 2021 40 2392 2402 10.1109/TMI.2021.3077484 33945474 · DOI
  18. [24]Nguyen, H. H., Saarakkala, S., Blaschko, M. B. & Tiulpin, A. CLIMAT: Clinically-Inspired Multi-Agent Transformers for Knee Osteoarthritis Trajectory Forecasting. in 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI) 1–5 (2022). PMC10880139 37672368
  19. [26]Huang S-C Pareek A Zamanian R Banerjee I Lungren MP Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: A case-study in pulmonary embolism detection Sci. Rep. 2020 10 22147 10.1038/s41598-020-78888-w 33335111 PMC7746687 · DOI · Paper page
  20. [27]Schulz S Multimodal deep learning for prognosis prediction in renal cancer Front. Oncol. 2021 11 788740 10.3389/fonc.2021.788740 34900744 PMC8651560 · DOI
  21. [28]Laleh, N. G. et al. Adversarial attacks and adversarial robustness in computational pathology. 2022.03.15.484515. 10.1101/2022.03.15.484515v1 (2022). PMC9522657 36175413 · DOI
  22. [29]Saldanha OL Swarm learning for decentralized artificial intelligence in cancer histopathology Nat. Med. 2022 28 1232 1239 10.1038/s41591-022-01768-5 35469069 PMC9205774 · DOI · Paper page
  23. [30]Lin, T.-Y., Goyal, P., Girshick, R., He, K. & Dollar, P. Focal loss for dense object detection. in 2980–2988 (2017). 30040631
  24. [31]Tu, Z. et al. MaxViT: Multi-axis vision transformer. in Computer Vision – ECCV 2022 (eds. Avidan, S., Brostow, G., Cissé, M., Farinella, G. M. & Hassner, T.) 459–479 (Springer Nature Switzerland, 2022).
  25. [33]Self, W. H., Courtney, D. M., McNaughton, C. D., Wunderink, R. G. & Kline, J. A. High discordance of chest x-ray and computed tomography for detection of pulmonary opacities in ED patients: implications for diagnosing pneumonia. Am. J. Emerg. Med.31, 401–405 (2013). PMC3556231 23083885
  26. [34]Johnson, A. E. W. et al. MIMIC-CXR-JPG, a Large publicly available database of labelled chest radiographs. Preprint at http://arxiv.org/abs/1901.07042 (2019). PMC6908718 31831740 · arXiv
  27. [35]Goldberger AL PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals Circulation 2000 101 E215 220 10.1161/01.CIR.101.23.e215 10851218 · DOI · Paper page
  28. [36]Deng, J. et al. ImageNet: A large-scale hierarchical image database. in 2009 IEEE Conference on Computer Vision and Pattern Recognition 248–255 (2009).
  29. [37]Harutyunyan H Khachatrian H Kale DC Ver Steeg G Galstyan A Multitask learning and benchmarking with clinical time series data Sci. Data 2019 6 96 10.1038/s41597-019-0103-9 31209213 PMC6572845 · DOI · Paper page
  30. [38]Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. Preprint at http://arxiv.org/abs/1711.05101 (2019). · arXiv
  31. [39]Loshchilov, I. & Hutter, F. SGDR: Stochastic gradient descent with warm restarts. Preprint at http://arxiv.org/abs/1608.03983 (2017). · arXiv
  32. [40]Konietschke F Pauly M Bootstrapping and permuting paired t-test type statistics Stat. Comput. 2014 24 283 296 10.1007/s11222-012-9370-4 · DOI
  33. [41]Unal I Defining an optimal cut-point value in ROC analysis: An alternative approach Comput. Math. Methods Med. 2017 2017 3762651 10.1155/2017/3762651 28642804 PMC5470053 · DOI · Paper page
  34. [42]Joze, H. R. V., Shaban, A., Iuzzolino, M. L. & Koishida, K. MMTM: Multimodal transfer module for CNN fusion. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 13289–13299 (2020).

Cited by

Sorted by citations

Authors
Khader, F., Kather, J. N., Müller-Franzes, G., Wang, T., Han, T., Tayebi Arasteh, S., Hamesch, K., Bressem, K., Haarburger, C., Stegmaier, J., Kuhl, C., Nebelung, S., & Truhn, D.
Affiliations
  • Universitätsklinikum Aachen (DE)
  • University of Leeds (GB)
  • RWTH Aachen University (DE)
  • Stiftung Charité (DE)
Corresponding author
Khader, Firas
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2023-07-01
Volume
13
Issue
1
Pages
10666-10666
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/s41598-023-37835-1
PMC
PMC10314902
OpenAlex
W4382797771
Cited by
57
References
7
Citations by year
2026: 21 · 2025: 19 · 2024: 16 · 2023: 1
FWCI
5.37
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • RWTH Publications (RWTH Aachen) (submittedVersion, other-oa)
Funders
RWTH Aachen University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
intensive care, chest radiograph, clinical data, transformer models, in-hospital mortality, prognosis, data integration, missing data robustness
MeSH terms
Critical Care, Diagnostic Imaging, Humans, Electric Power Supplies, Retrospective Studies, Area Under Curve

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review