The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index; 16 of the 23 sources its claims cite were checked for support.
Integrated multimodal artificial intelligence framework for healthcare applications
Open accessAbstract
Artificial intelligence (AI) systems hold great promise to improve healthcare over the next decades. Specifically, AI systems leveraging multiple data sources and input modalities are poised to become a viable method to deliver more accurate results and deployable pipelines across a wide range of applications. In this work, we propose and evaluate a unified Holistic AI in Medicine (HAIM) framework to facilitate the generation and testing of AI systems that leverage multimodal inputs. Our approach uses generalizable data pre-processing and machine learning modeling stages that can be readily adapted for research and deployment in healthcare environments. We evaluate our HAIM framework by training and characterizing 14,324 independent models based on HAIM-MIMIC-MM, a multimodal clinical database ( N = 34,537 samples) containing 7279 unique hospitalizations and 6485 patients, spanning all possible input combinations of 4 data modalities (i.e., tabular, time-series, text, and images), 11 unique data sources and 12 predictive tasks. We show that this framework can consistently and robustly produce models that outperform similar single-source approaches across various healthcare demonstrations (by 6–33%), including 10 distinct chest pathology diagnoses, along with length-of-stay and 48 h mortality predictions. We also quantify the contribution of each modality and data source using Shapley values, which demonstrates the heterogeneity in data modality importance and the necessity of multimodal inputs across different healthcare-relevant tasks. The generalizable properties and flexibility of our Holistic AI in Medicine (HAIM) framework could offer a promising pathway for future multimodal predictive systems in clinical and operational healthcare settings.
Overview
CoChat Summary
## TL;DR
The paper proposes HAIM, a modular multimodal AI framework for healthcare that turns heterogeneous EHR data into fixed-size patient embeddings for downstream prediction. On a multimodal MIMIC-based dataset, the framework trained 14,324 XGBoost models and reported consistent AUROC gains over single-source baselines across chest X-ray pathology, length-of-stay, and 48 h mortality tasks.
## Key findings
- The compiled dataset, HAIM-MIMIC-MM, contained 34,537 samples, 7,279 hospitalization stays, and 6,485 unique patients.
- Across all experiments, the framework evaluated 14,324 independent models: 10,230 for chest pathology diagnosis, 2,047 for length-of-stay, and 2,047 for 48 h mortality prediction.
- Multimodal/multi-source inputs improved average performance over canonical single-source systems by 6–22% for chest X-ray pathology prediction, 8–20% for length-of-stay, and 11–33% for 48 h mortality.
- For chest pathology prediction, the minimum per-task AUROC improvements were reported as: Fracture 6%, Lung Lesion 7%, Enlarged cardiomediastinum 9%, Consolidation 10%, Pneumonia 8%, Atelectasis 6%, Lung opacity 7%, Pneumothorax 8%, Edema 10%, and Cardiomegaly 10%.
- Shapley-value analysis showed that vision data contributed most to chest pathology tasks, while historical time-series records were most relevant for length-of-stay and 48 h mortality.
## Methods
- The study built a unified multimodal dataset by combining MIMIC-IV v1.0 and MIMIC-CXR-JPG v2.0.0 using matched patient/admission/imaging identifiers.
- Each patient admission was converted into a patient-centric file and embedded across 4 modalities and 11 data sources: tabular demographics, time-series events, text notes, and images; fixed embeddings were extracted using Clinical BERT for text and Densenet121-res224-chex for images.
- Downstream tasks included 10 chest pathology binary classifications, length-of-stay (discharged alive within 48 h vs otherwise), and 48 h mortality (expired within 48 h vs otherwise). Models were XGBoost, trained with 80/20 splits, repeated 5 times, with 5-fold cross-validated grid search on the training set; performance was reported by test AUROC.
## Limitations
- The paper states that Shapley analysis can reveal small or negative contributions when there is absence of extracted information or error propagation, and that such issues may be addressed by removing modalities or changing feature extractors.
- The paper notes that some improvements showed diminishing returns and local non-monotonic effects due to data source availability, error, and redundancy.
- Evident from the text: the demonstration is based on a single source dataset from MIMIC/BIDMC, so generalization to other hospitals or EHR systems is not directly established by the reported experiments.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
10 Valid · 5 Not valid · 3 Not checked · 58 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Such anticipated utility, particularly in AI/ML systems aimed to improve clinical efficiency and patient outcomes, will require knowledge from multiple data sources and various input modalities2–4.”
§ Integrated multimodal artificial intelligence framework for healthcare applications > Introduction
What the sources say
- Partly supportedcites [2] Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I. & Lungren, M. P. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Dig. Med. 3, 1–9 (2020). sourceExperimental — accuracy not yet measured
The Integration section describes medical data from records, imaging, lab results and wearables as potential inputs for AI analysis. The Conclusion links access across independent systems and varied data types to machine-learning strategies. These passages support the value of multiple sources and modalities, but do not establish that AI/ML systems require them to improve clinical efficiency or patient outcomes. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites [3] Gietzelt, M., Löpprich, M., Karmen, C. & Ganzinger, M. Models and data sources used in systems medicine. Methods Inf. Med. 55, 107–113 (2016). source
- Partly supportedcites [4] Boonn, W. W. & Langlotz, C. P. Radiologist use of and perceived need for patient data access. J. Dig. imaging 22, 357–362 (2009). sourceExperimental — accuracy not yet measured
The Conclusion calls for an application that retrieves pertinent patient information from multiple clinical systems and presents it efficiently to radiologists, with expected benefits for interpretation and quality of care. The Results identify several useful information sources. The paper does not establish that AI/ML systems require multiple input modalities; that part extends beyond its discussion of radiologists’ access to clinical data. (LLM, conf=0.86)
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For this work, we utilize the Medical Information Cart for Intensive Care (MIMIC)-IV32,33, an openly accessible database that contains de-identified records of 383,220 individual patients admitted to the ICU or emergency department (ED) of Beth Israel Deaconess Medical Center (BIDMC) in Boston, MA, USA, between 2008 and 2019 (inclusive).”
§ Integrated multimodal artificial intelligence framework for healthcare applications > Methods > Dataset
What the sources say
- Partly supportedcites [32] Johnson, A. et al. MIMIC-IV (version 1.0). PhysioNet, https://doi.org/10.13026/s6n6-xd98. (2021). sourceExperimental — accuracy not yet measured
The Methods (page unavailable) confirm that MIMIC-IV draws on de-identified records of BIDMC ICU or emergency-department patients from 2008–2019. The copy does not substantiate the figure of 383,220 individual patients. Its access information also describes the database as restricted to credentialed users who complete training and sign a data-use agreement, rather than openly accessible without qualification. (LLM, conf=0.87)
- Not checked — no full text of the sourcecites [33] Goldberger, A. L. et al. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation 101, e215–e220 (2000). source
plainn = 383,220- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, developing unified and scalable pipelines that can consistently be applied to train multimodal AI/ML systems that leverage and outperform their single-modality counterparts has remained challenging2.”
§ Integrated multimodal artificial intelligence framework for healthcare applications > Introduction
What the sources say
- Could not verifycites [2] Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I. & Lungren, M. P. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Dig. Med. 3, 1–9 (2020). sourceExperimental — accuracy not yet measured
The Conclusion (page not provided) describes a scalable data lake that can store varied healthcare data and support machine learning; the Discussion describes challenges in consolidating medical-record systems. Neither addresses the claimed difficulty of building unified pipelines to train multimodal AI/ML models that outperform single-modality models. Searches for multimodal performance and training-pipeline challenges found no passage on that point. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Furthermore, it has been shown that multimodality in most of these domains can increase the performance of AI/ML systems (accuracy: 1.2–27.7%) compared to single-modality approaches for the same task2.”
§ Integrated multimodal artificial intelligence framework for healthcare applications > Introduction
What the sources say
- Could not verifycites [2] Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I. & Lungren, M. P. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Dig. Med. 3, 1–9 (2020). sourceExperimental — accuracy not yet measured
The Integration and Discussion sections discuss combining health data from different sources for possible AI analysis, but neither reports a comparison with single-modality systems. Searches for multimodality, comparative AI/ML performance, and the 1.2–27.7% accuracy range found no supporting passage in our copy; page numbers were unavailable. (LLM, conf=0.76)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Multimodal architectures for AI/ML systems are attractive because they can emulate the input conditions that clinicians and healthcare administrators currently use to perform predictions and respond to their complex decision-making landscape2,5.”
§ Integrated multimodal artificial intelligence framework for healthcare applications > Introduction
What the sources say
- Partly supportedcites [2] Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I. & Lungren, M. P. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Dig. Med. 3, 1–9 (2020). sourceExperimental — accuracy not yet measured
The Integration and Discussion sections describe combining varied health data, including records and wearables, for AI analysis and physicians’ treatment planning. The Conclusion says the proposed data system could support machine-learning strategies. The paper does not say that multimodal AI/ML architectures emulate clinicians’ or administrators’ existing inputs for prediction and complex decision-making; its architecture is for blockchain-based record management. (LLM, conf=0.84)
- Partly supportedcites [5] Wang, W. & Krishnan, E. Big data and clinicians: a review on the state of the science. JMIR Med. Inform. 2, e1 (2014). sourceExperimental — accuracy not yet measured
The Results section (Method Development Studies; page unavailable) describes tools that use clinical data to aid analysis and treatment decisions. The Discussion (Methods for Manipulation of Clinical Big Data; page unavailable) also describes diverse data and data integration. Neither passage says that multimodal AI/ML architectures emulate the inputs clinicians or healthcare administrators use, so the claim’s central rationale is unsupported. (LLM, conf=0.78)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
31 more claims · 1 with their cited sources checked
Create a free account to see all 76 claims and every source verdict.
- It may state two separate claims as one. 21
- Probably not a substantial claim. 24
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“MIMIC-IV-MM is generated by joining MIMIC-IV, MIMIC-CXR, and MIMIC-IV-Note on the triplet of patient subject ID, hospital admission ID, and ICU-stay ID.”
From Multimodal risk prediction with physiological signals, medical images and clinical notes (2024) · cites this paper as [25]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These approaches combine heterogeneous data sources to improve diagnostic accuracy and contextual understanding in complex clinical scenarios [9].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In dermatology, ophthalmology, and radiology, convolutional neural networks have reported performance comparable to specialists in selected controlled tasks, such as skin cancer classification, diabetic retinopathy detection, breast cancer screening, and lung nodule identification [9,12].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These systems may improve triage prioritisation, reduce waiting times, and support resource allocation [9,15].”
From Artificial Intelligence in Clinical Decision-Making: A Comprehensive Review of Diagnostic, Prognostic, and Therapeutic Applications, Validation Gaps, Bias, and Deployment Challenges (2026) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When the authors tested their framework in over 14’000 different prediction models, they found that predictions based on multimodal data outperform unimodal comparators by 6-30%.”
From Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging (2024) · cites this paper as [16]
Experimental — accuracy not yet measured
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809
Re-epithelialization and immune cell behaviour in an ex vivo human skin model
Rakita, A., Nikolić, N., Mildner, M., Matiasek, J., & Elbe-Bürger, A. (2020) · Scientific Reports · doi:10.1038/s41598-019-56847-4 · Cited by 8,228
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 1×
“0) improves on MIMIC-III 34 to provide public access to the EHR data of over 40,000 hospitalized patients based on the BIDMC ’ s MetaVision clinical information system.”
MethodU1 snRNP regulates cancer cell migration and invasion in vitro
Oh, J.-M., Venters, C. C., Di, C., Pinto, A. M., Wan, L., Younis, I., Cai, Z., Arai, C., So, B. R., Duan, J., et al. (2020) · Nature Communications · doi:10.1038/s41467-019-13993-7 · Cited by 7,157
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648
Explaining prediction models and individual predictions with feature contributions
Štrumbelj, E., & Kononenko, I. (2014) · Knowledge and Information Systems · doi:10.1007/s10115-013-0679-x · Cited by 2,286
Scalable and accurate deep learning with electronic health records.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · eScholarship (California Digital Library) · Cited by 2,242
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · cited 4×
“We obtained embeddings for the unstructured free text (E radn , E ecgn , and E econ ) by concatenating all available text from each of these types of notes as continuous strings and then by processing them using Clinical BERT 39 , a transformer-based bidirectional encoder model pre-trained on a large corpus of biomedi…”
MethodInfluentialA clinically applicable approach to continuous prediction of future acute kidney injury
Tomašev, N., Glorot, X., Rae, J. W., Zielinski, M., Askham, H., Saraiva, A., Mottram, A., Meyer, C., Ravuri, S., Protsyuk, I., et al. (2019) · Nature · doi:10.1038/s41586-019-1390-1 · Cited by 1,086
Research works printed but not linked: 1
- Royalty, J. P. Machine Learning Time-to-Event Mortality Prediction in MIMIC-IV Critical Care Database (Doctoral dissertation). Undergraduate Research Scholars Program. Available electronically from https://hdl.handle.net/1969.1/194429 (2021).
Cited by
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324
Review of multimodal machine learning approaches in healthcare
Krones, F., Marikkar, U., Parsons, G., Szmul, A., & Mahdi, A. (2025) · Information Fusion · doi:10.1016/j.inffus.2024.102690 · Cited by 235
Singh, A. V., Chandrasekar, V., Paudel, N., Laux, P., Luch, A., Gemmati, D., Tisato, V., Prabhu, K. S., Shahab Uddin, M. S., & Dakua, S. P. (2023) · Biomedicine & Pharmacotherapy · doi:10.1016/j.biopha.2023.114784 · Cited by 167
“Reproduced with permission[21].”
Lin, M., Guo, J., Gu, Z., Tang, W., Tao, H., You, S., Jia, D., Sun, Y., & Jia, P. (2025) · Journal of Translational Medicine · doi:10.1186/s12967-025-06425-2 · Cited by 175
“…the integration of patient cardiac ultrasound and medical history with omics data requires not only consideration of the effects of platform noise, but also the use of autoencoders or deep generative models for feature extraction and dimensionality reduction of high-dimensional omics data [109].”
Integrating multi-omics data with EHR for precision medicine using advanced artificial intelligence
Tong, L., Shi, W., Isgut, M., Zhong, Y., Lais, P., Gloster, L., Sun, J., Swain, A., Giuste, F., & Wang, M. D. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2023.3324264 · Cited by 130
Metadata
- Authors
- Soenksen, L. R., Ma, Y., Zeng, C., Boussioux, L., Villalobos Carballo, K., Na, L., Wiberg, H. M., Li, M. L., Fuentes, I., & Bertsimas, D.
- Affiliations
- Harvard University (US)
- Massachusetts Institute of Technology (US)
- Corresponding author
- Bertsimas, Dimitris
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2022-09-20
- Volume
- 5
- Issue
- 1
- Pages
- 149-149
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC9489871
- OpenAlex
- W4296776307
- Cited by
- 380
- References
- 47
- Citations by year
- 2026: 141 · 2025: 129 · 2024: 74 · 2023: 33 · 2022: 3
- FWCI
- 19.74
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (174530)
- Harvard T.H. Chan School of Public Health
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- multimodal AI, multimodal medical data, length of stay prediction, Shapley value, hospitalization data, clinical prediction model
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Integrated multimodal artificial intelligence framework for healthcare applications. https://app.cochat.ai/papers/10.1038/s41746-022-00689-4
- The paper
- Soenksen, L. R., Ma, Y., Zeng, C., Boussioux, L., Villalobos Carballo, K., Na, L., Wiberg, H. M., Li, M. L., Fuentes, I., & Bertsimas, D. (2022). Integrated multimodal artificial intelligence framework for healthcare applications. npj Digital Medicine. https://doi.org/10.1038/s41746-022-00689-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.