Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index; 16 of the 23 sources its claims cite were checked for support.

Integrated multimodal artificial intelligence framework for healthcare applications

Open access
Soenksen, Luis R., Ma, Yu, Zeng, Cynthia, Boussioux, Leonard, Villalobos Carballo, Kimberly, Na, Liangyuan, Wiberg, Holly M., Li, Michael L., Fuentes, Ignacio, Bertsimas, Dimitris · 2022 · npj Digital Medicine

Abstract

Artificial intelligence (AI) systems hold great promise to improve healthcare over the next decades. Specifically, AI systems leveraging multiple data sources and input modalities are poised to become a viable method to deliver more accurate results and deployable pipelines across a wide range of applications. In this work, we propose and evaluate a unified Holistic AI in Medicine (HAIM) framework to facilitate the generation and testing of AI systems that leverage multimodal inputs. Our approach uses generalizable data pre-processing and machine learning modeling stages that can be readily adapted for research and deployment in healthcare environments. We evaluate our HAIM framework by training and characterizing 14,324 independent models based on HAIM-MIMIC-MM, a multimodal clinical database ( N = 34,537 samples) containing 7279 unique hospitalizations and 6485 patients, spanning all possible input combinations of 4 data modalities (i.e., tabular, time-series, text, and images), 11 unique data sources and 12 predictive tasks. We show that this framework can consistently and robustly produce models that outperform similar single-source approaches across various healthcare demonstrations (by 6–33%), including 10 distinct chest pathology diagnoses, along with length-of-stay and 48 h mortality predictions. We also quantify the contribution of each modality and data source using Shapley values, which demonstrates the heterogeneity in data modality importance and the necessity of multimodal inputs across different healthcare-relevant tasks. The generalizable properties and flexibility of our Holistic AI in Medicine (HAIM) framework could offer a promising pathway for future multimodal predictive systems in clinical and operational healthcare settings.

Overview

CoChat Summary

## TL;DR

The paper proposes HAIM, a modular multimodal AI framework for healthcare that turns heterogeneous EHR data into fixed-size patient embeddings for downstream prediction. On a multimodal MIMIC-based dataset, the framework trained 14,324 XGBoost models and reported consistent AUROC gains over single-source baselines across chest X-ray pathology, length-of-stay, and 48 h mortality tasks.

## Key findings

  • The compiled dataset, HAIM-MIMIC-MM, contained 34,537 samples, 7,279 hospitalization stays, and 6,485 unique patients.
  • Across all experiments, the framework evaluated 14,324 independent models: 10,230 for chest pathology diagnosis, 2,047 for length-of-stay, and 2,047 for 48 h mortality prediction.
  • Multimodal/multi-source inputs improved average performance over canonical single-source systems by 6–22% for chest X-ray pathology prediction, 8–20% for length-of-stay, and 11–33% for 48 h mortality.
  • For chest pathology prediction, the minimum per-task AUROC improvements were reported as: Fracture 6%, Lung Lesion 7%, Enlarged cardiomediastinum 9%, Consolidation 10%, Pneumonia 8%, Atelectasis 6%, Lung opacity 7%, Pneumothorax 8%, Edema 10%, and Cardiomegaly 10%.
  • Shapley-value analysis showed that vision data contributed most to chest pathology tasks, while historical time-series records were most relevant for length-of-stay and 48 h mortality.

## Methods

  • The study built a unified multimodal dataset by combining MIMIC-IV v1.0 and MIMIC-CXR-JPG v2.0.0 using matched patient/admission/imaging identifiers.
  • Each patient admission was converted into a patient-centric file and embedded across 4 modalities and 11 data sources: tabular demographics, time-series events, text notes, and images; fixed embeddings were extracted using Clinical BERT for text and Densenet121-res224-chex for images.
  • Downstream tasks included 10 chest pathology binary classifications, length-of-stay (discharged alive within 48 h vs otherwise), and 48 h mortality (expired within 48 h vs otherwise). Models were XGBoost, trained with 80/20 splits, repeated 5 times, with 5-fold cross-validated grid search on the training set; performance was reported by test AUROC.

## Limitations

  • The paper states that Shapley analysis can reveal small or negative contributions when there is absence of extracted information or error propagation, and that such issues may be addressed by removing modalities or changing feature extractors.
  • The paper notes that some improvements showed diminishing returns and local non-monotonic effects due to data source availability, error, and redundancy.
  • Evident from the text: the demonstration is based on a single source dataset from MIMIC/BIDMC, so generalization to other hospitals or EHR systems is not directly established by the reported experiments.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-022-00689-4

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Holistic Artificial Intelligence in Medicine (HAIM) framework. Under this framework, databases and tables sourced from specific healthcare institutions such as HAIM-MIMIC-MM combined from MIMIC-IV and MIMIC-CXR-JPG for this work are processed to generate individual patient files. These files contain past and present multimodal patient information from the moment of admission. For processing under the HAIM framework, every data modality is fed to independent embedding generating streams. In this work, tabular data is minimally processed using simple transformations or normalizations to produce encodings or embedding-like categorical numerical values (ETabular(n,t), where n = unique stay/hospitalization/patient and t = sampling time). Selected time-series are processed by generating statistical metrics on each of the time-dependent signals to produce embeddings representative of their trends (ETimeSeries(n,t)) from the moment of admission until the sampling time. Natural language inputs such as notes are processed using a pre-trained transformer neural network to generate text embeddings of fixed size (EText(n,t)). Furthermore, image inputs such as X-rays are processed using a pre-trained convolutional neural network to also extract fixed-size embeddings out of the model output probability vectors and dense features (EImages(n,t)). While not done in this work, thanks to the modularity of the embedding extraction process in the HAIM framework, other pre-trained models or systems could be added to generate embeddings from other types of data sources if needed (EOther(n,t)). All generated embeddings are concatenated to generate a fusion embedding, which can be used to train, test, and deploy models for predictive analytics in healthcare operations. For this work, we tested and utilized only XGBoost as a canonical type of architecture for building the downstream predictive models based on fusion embeddings. CNN Convolutional Neural Network, CT Computerized Tomography, ECG Electrocardiogram, ECO Echocardiogram, MRI Magnetic Resonance Imaging, NN Neural Network, O2 Oxygen, ReLU Rectified Linear Unit, RNN Recurrent Neural Network, US Ultrasound.
Fig. 1 Holistic Artificial Intelligence in Medicine (HAIM) framework. Under this framework, databases and tables sourced from specific healthcare institutions such as HAIM-MIMIC-MM combined from MIMIC-IV and MIMIC-CXR-JPG for this work are processed to generate individual patient files. These files contain past and present multimodal patient information from the moment of admission. For processing under the HAIM framework, every data modality is fed to independent embedding generating streams. In this work, tabular data is minimally processed using simple transformations or normalizations to produce encodings or embedding-like categorical numerical values (ETabular(n,t), where n = unique stay/hospitalization/patient and t = sampling time). Selected time-series are processed by generating statistical metrics on each of the time-dependent signals to produce embeddings representative of their trends (ETimeSeries(n,t)) from the moment of admission until the sampling time. Natural language inputs such as notes are processed using a pre-trained transformer neural network to generate text embeddings of fixed size (EText(n,t)). Furthermore, image inputs such as X-rays are processed using a pre-trained convolutional neural network to also extract fixed-size embeddings out of the model output probability vectors and dense features (EImages(n,t)). While not done in this work, thanks to the modularity of the embedding extraction process in the HAIM framework, other pre-trained models or systems could be added to generate embeddings from other types of data sources if needed (EOther(n,t)). All generated embeddings are concatenated to generate a fusion embedding, which can be used to train, test, and deploy models for predictive analytics in healthcare operations. For this work, we tested and utilized only XGBoost as a canonical type of architecture for building the downstream predictive models based on fusion embeddings. CNN Convolutional Neural Network, CT Computerized Tomography, ECG Electrocardiogram, ECO Echocardiogram, MRI Magnetic Resonance Imaging, NN Neural Network, O2 Oxygen, ReLU Rectified Linear Unit, RNN Recurrent Neural Network, US Ultrasound.CC BY · PubMed Central
Fig. 2 Performance of the multimodal HAIM framework on various demonstrations for healthcare operations. a Average and standard deviation values of the area under the receiver operating characteristic (AUROC) for all demonstrations including pathology diagnosis (i.e., lung lesions, fractures, atelectasis, lung opacities, pneumothorax, enlarged cardio mediastinum, cardiomegaly, pneumonia, consolidation, and edema), as well as length-of-stay and 48 h mortality prediction. The number of modalities refers to the coverage among tabular, time-series, text, and image data. The number of sources refers to the coverage among available input data sources (10 for pathology diagnosis, while 11 for length-of-stay and 48 h mortality prediction). Thus, the position (Modality = 2, Sources = 3) corresponds to the average AUROC of all models across all input combinations covering any 2 modalities using any 3 input sources. Increasing gradients on average AUROC appear to follow from increasing the number of modalities and number of sources across all evaluated tasks. Decreasing gradients on AUROC standard deviations follow from less variability in performance as a higher number of modalities and data sources is used. b Receiver operating characteristic (ROC) curves for typical HAIM model across all use cases exhibiting input multimodal. c ROC curves for a best-performing model with single-modality inputs across the same use cases. Consistent averaged improvements across all tasks are observed in multimodality as compared to single-modality systems. AUROC Area under the curve, AUROC Area under the receiver operating characteristic curve, CM Cardiomediastinum. Dx Diagnosis, HAIM Holistic Artificial Intelligence in Medicine, Ops Operations, SD Standard deviation.
Fig. 2 Performance of the multimodal HAIM framework on various demonstrations for healthcare operations. a Average and standard deviation values of the area under the receiver operating characteristic (AUROC) for all demonstrations including pathology diagnosis (i.e., lung lesions, fractures, atelectasis, lung opacities, pneumothorax, enlarged cardio mediastinum, cardiomegaly, pneumonia, consolidation, and edema), as well as length-of-stay and 48 h mortality prediction. The number of modalities refers to the coverage among tabular, time-series, text, and image data. The number of sources refers to the coverage among available input data sources (10 for pathology diagnosis, while 11 for length-of-stay and 48 h mortality prediction). Thus, the position (Modality = 2, Sources = 3) corresponds to the average AUROC of all models across all input combinations covering any 2 modalities using any 3 input sources. Increasing gradients on average AUROC appear to follow from increasing the number of modalities and number of sources across all evaluated tasks. Decreasing gradients on AUROC standard deviations follow from less variability in performance as a higher number of modalities and data sources is used. b Receiver operating characteristic (ROC) curves for typical HAIM model across all use cases exhibiting input multimodal. c ROC curves for a best-performing model with single-modality inputs across the same use cases. Consistent averaged improvements across all tasks are observed in multimodality as compared to single-modality systems. AUROC Area under the curve, AUROC Area under the receiver operating characteristic curve, CM Cardiomediastinum. Dx Diagnosis, HAIM Holistic Artificial Intelligence in Medicine, Ops Operations, SD Standard deviation.CC BY · PubMed Central
Fig. 3 Multimodal HAIM framework is a flexible and robust method to improve predictive capacity for healthcare machine learning systems as compared to single-modality approaches. a Average percent change of area under the receiver operating characteristic curve (Avg. ΔAUROC) for all tested multimodality HAIM models as compared to their single-source single-modality counterparts. While different models exhibit varying degrees of improvement, all tested models show positive Avg. ΔAUROC percentages. The number of modalities refers to the coverage among tabular, time-series, text, and image data. The number of sources refers to the coverage among available input data sources (10 for pathology diagnosis, 11 for length-of-stay, and 48 h mortality prediction). Thus, the position (Modality = 2, Sources = 3) corresponds to the average AUROC of all models across all input combinations covering any 2 modalities using any 3 input sources. b Expanded Avg. ΔAUROC percentages for all tested multimodality HAIM models and ordered by the number of used modalities (i.e., tabular, time-series, text, or images) as well as the number of used data sources. c Waterfall plots of aggregated Shapley values for independent data modalities per predictive task. While Shapley values for all data modalities appear to be positively contributing to the predictive capacity of all models, different tasks exhibit distinct distributions of aggregated Shapley values. d High-level schematic of the HAIM pipeline developed to support the presented work. After data collection or sourcing (HAIM-MIMIC-MM for this work), a process of feature selection and embedding extraction is applied to feed fusion embeddings into a process of iterative architecture engineering (model and hyperparameter selection). After particular models are selected and trained, they can be benchmarked to test and report results. This process concludes by the selection of a model for deployment in a use case scenario.
Fig. 3 Multimodal HAIM framework is a flexible and robust method to improve predictive capacity for healthcare machine learning systems as compared to single-modality approaches. a Average percent change of area under the receiver operating characteristic curve (Avg. ΔAUROC) for all tested multimodality HAIM models as compared to their single-source single-modality counterparts. While different models exhibit varying degrees of improvement, all tested models show positive Avg. ΔAUROC percentages. The number of modalities refers to the coverage among tabular, time-series, text, and image data. The number of sources refers to the coverage among available input data sources (10 for pathology diagnosis, 11 for length-of-stay, and 48 h mortality prediction). Thus, the position (Modality = 2, Sources = 3) corresponds to the average AUROC of all models across all input combinations covering any 2 modalities using any 3 input sources. b Expanded Avg. ΔAUROC percentages for all tested multimodality HAIM models and ordered by the number of used modalities (i.e., tabular, time-series, text, or images) as well as the number of used data sources. c Waterfall plots of aggregated Shapley values for independent data modalities per predictive task. While Shapley values for all data modalities appear to be positively contributing to the predictive capacity of all models, different tasks exhibit distinct distributions of aggregated Shapley values. d High-level schematic of the HAIM pipeline developed to support the presented work. After data collection or sourcing (HAIM-MIMIC-MM for this work), a process of feature selection and embedding extraction is applied to feed fusion embeddings into a process of iterative architecture engineering (model and hyperparameter selection). After particular models are selected and trained, they can be benchmarked to test and report results. This process concludes by the selection of a model for deployment in a use case scenario.CC BY · PubMed Central

Verification

Claims the paper makes

10 Valid · 5 Not valid · 3 Not checked · 58 Own results and interpretations

31 more claims · 1 with their cited sources checked

Create a free account to see all 76 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 21
  • Probably not a substantial claim. 24
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

4 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 47 references

Research works printed but not linked: 1

  1. Royalty, J. P. Machine Learning Time-to-Event Mortality Prediction in MIMIC-IV Critical Care Database (Doctoral dissertation). Undergraduate Research Scholars Program. Available electronically from https://hdl.handle.net/1969.1/194429 (2021).

Cited by

See all 380

Metadata

Authors
Soenksen, L. R., Ma, Y., Zeng, C., Boussioux, L., Villalobos Carballo, K., Na, L., Wiberg, H. M., Li, M. L., Fuentes, I., & Bertsimas, D.
Affiliations
  • Harvard University (US)
  • Massachusetts Institute of Technology (US)
Corresponding author
Bertsimas, Dimitris
Venue
npj Digital Medicine
Published
2022-09-20
Volume
5
Issue
1
Pages
149-149
ISSN
2398-6352 (ISSN-L)
OpenAlex
W4296776307
Cited by
380
References
47
Citations by year
2026: 141 · 2025: 129 · 2024: 74 · 2023: 33 · 2022: 3
FWCI
19.74
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (174530)
  • Harvard T.H. Chan School of Public Health
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
multimodal AI, multimodal medical data, length of stay prediction, Shapley value, hospitalization data, clinical prediction model

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Integrated multimodal artificial intelligence framework for healthcare applications. https://app.cochat.ai/papers/10.1038/s41746-022-00689-4
The paper
Soenksen, L. R., Ma, Y., Zeng, C., Boussioux, L., Villalobos Carballo, K., Na, L., Wiberg, H. M., Li, M. L., Fuentes, I., & Bertsimas, D. (2022). Integrated multimodal artificial intelligence framework for healthcare applications. npj Digital Medicine. https://doi.org/10.1038/s41746-022-00689-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.