The DOI matches its Crossref record; 55 of the 60 references it cites were found in a registry or an index; 38 of the 90 sources its claims cite were checked for support.
Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines
Open accessAbstract
In this paper, we describe different data fusion techniques that can be applied to combine medical imaging with EHR, and systematically review medical data fusion literature published between 2012 and 2020. We conducted a systematic search on PubMed and Scopus for original research articles leveraging deep learning for fusion of multimodality data. In total, we screened 985 studies and extracted data from 17 papers. By means of this systematic review, we present current knowledge, summarize important results and provide implementation guidelines to serve as a reference for researchers interested in the application of multimodal data fusion.
Overview
CoChat Summary
TL;DR
This systematic review consolidates deep learning approaches that fuse medical imaging with electronic health record (EHR) data, defining early, joint, and late fusion strategies and reviewing 17 eligible studies published between 2012 and 2020. It concludes that multimodal fusion generally improves performance over single-modality models, though no single fusion strategy is consistently optimal, and provides implementation guidelines for future work.
Key findings
- From 985 screened studies, 44 underwent full-text screening and 17 met eligibility criteria; early fusion was used in 11/17 studies, joint fusion in 4/17, and late fusion in 3/17.
- Across reviewed studies, multimodal fusion generally improved performance over single-modality models, with reported accuracy gains of 1.2–27.7% and AUROC gains of 0.02–0.16.
- No single fusion strategy consistently led to optimal performance across all clinical domains; examples include Kharazmi et al. achieving 91.1% accuracy with fusion versus 84.7% for dermoscopic images and 75.6% for patient profile, and Yala et al. achieving 0.70 AUROC with joint fusion versus 0.68 for mammograms and 0.67 for risk scores.
- Non-imaging EHR data in the included studies were limited, mostly to basic demographics, a narrow range of categorical clinical history, or disease-specific features such as APOE4 or PSA, rather than feature-rich or time-series EHR data.
Methods
- PRISMA-based systematic review with searches in PubMed and Scopus for papers published 2012–2020, using key themes of “deep learning,” “multimodality fusion,” and “medical imaging”; segmentation, registration, and reconstruction terms were exclusion criteria, and preprints were excluded.
- Inclusion required original research applying deep learning for clinical outcome prediction or diagnosis by fusing medical imaging with EHR data; two independent researchers screened records using Covidence, with disagreements resolved by discussion and a third researcher if needed.
- Data extracted from each included study included fusion strategy, year, authors, clinical domain, target outcome, fusion details, imaging modality, non-imaging modality, number of samples, and model performance; AUROC was extracted when reported, otherwise accuracy, using the best-performing fusion experiment.
Limitations
- The paper states that its search excluded pre-2012 papers, that publication bias may overrepresent fusion advantages, that it only covered clinical prediction and diagnosis rather than segmentation or registration, and that heterogeneity across studies prevented meta-analysis; it also notes that some studies lacked independent test sets and that classifying unconventional fusion implementations introduced subjectivity.
- The paper states that the limited number of studies per medical field and heterogeneity made qualitative comparison difficult.
- Evident from the text: the review includes only 17 studies, and the non-imaging EHR data used across those studies were narrow, limiting conclusions about fusion with richer or time-series clinical data.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
19 Valid · 3 Not valid · 15 Not checked · 39 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In a survey of radiologists, the majority (87%) stated that clinical information had a significant impact on interpretation3.”
§ Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines > Introduction
What the sources say
- Partly supportedcites [3] Boonn, W. W. & Langlotz, C. P. Radiologist use of and perceived need for patient data access. J. Digit. Imaging 22, 357–362 (2009). sourceExperimental — accuracy not yet measured
The Results report that more than 87% of surveyed radiologists considered additional clinical information important and said it could change or modify their interpretive report. Table 1 records 87.0% as willing to modify or change a diagnosis based on such information. These are judgments about its potential impact, not reports that it had already significantly affected their interpretations. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“When the different input modalities have very different numbers of features, predictions might be overly influenced by the most feature-rich modality (e.g., Reda et al.40).”
§ Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines > Discussion > Implementation guidelines for fusion models
What the sources say
- Could not verifycites Reda et al.40 Reda, I. et al. Deep learning role in early diagnosis of prostate cancer. Technol. Cancer Res. Treat. 17, 153303461877553 (2018). sourceExperimental — accuracy not yet measured
The Imaging Features and Clinical Biomarkers and Experimental Results sections describe combining low-dimensional PSA data with DW-MRI features through diagnostic probabilities. Neither section says that a modality with more features might disproportionately influence predictions; searches for feature-count imbalance and modality dominance found no such passage. Page numbers were not available in our copy. (LLM, conf=0.73)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“Fusion approaches in other less complex clinical applications also improved performance over single modality models, even those in which single modality models have been widely reported to achieve high performance, such as pixel-based models for automated skin cancer detection43.”
§ Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines > Discussion
What the sources say
- Supportedcites [43] Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017). source
Split into 2 atoms — 2 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“while physicians struggle with accurate and reliable diagnostics even when multimodality is present, as proven by histopathological correlation42.”
§ Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines > Discussion
What the sources say
- Partly supportedcites [42] Beach, T. G., Monsell, S. E., Phillips, L. E. & Kukull, W. Accuracy of the Clinical Diagnosis of Alzheimer Disease at National Institute on Aging Alzheimer Disease Centers, 2005–2010. J. Neuropathol. Exp. Neurol. 71, 266–273 (2012). sourceExperimental — accuracy not yet measured
The Results (Diagnostic Classification Comparison; page unavailable) show imperfect agreement between clinical Alzheimer disease diagnoses and neuropathological findings, supporting the point that clinicians can have difficulty diagnosing AD accurately against a histopathological standard. The Materials and Methods (Analysis Strategy; page unavailable) do not establish that the clinical diagnoses used multiple diagnostic modalities, and the study does not assess diagnostic reliability. Its findings concern AD at participating disease centers, not physicians’ diagnostics generally. (LLM, conf=0.88)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“The importance of clinical context for accurate interpretation of imaging data is not limited to radiology; instead many other imaging-based medical specialties such as pathology, ophthalmology, and dermatology, also rely on clinical data to guide image interpretation in practice4–6.”
§ Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines > Introduction
What the sources say
- Not checked — no full text of the sourcecites [4] Comfere, N. I. et al. Provider-to-provider communication in dermatology and implications of missing clinical information in skin biopsy requisition forms: a systematic review. Int. J. Dermatol. 53, 549–557 (2014). source
- Supportedcites [5] Comfere, N. I. et al. Dermatopathologists’ concerns and challenges with clinical information in the skin biopsy requisition form: a mixed-methods study: Clinical information in dermatopathology. J. Cutan. Pathol. 42, 333–345 (2015). source
The Discussion says clinical information is crucial for accurate dermatopathologic interpretation and emphasizes the importance of clinical context. The Results report that most respondents considered the quality of clinical information to have a large impact on diagnostic accuracy. This supports the dermatopathology example to which this source is pertinent. (LLM, conf=0.91)
- Not checked — no full text of the sourcecites [6] Jonas, J. B. et al. Glaucoma. The Lancet 390, 2183–2193 (2017). source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
38 more claims · 9 with their cited sources checked
Create a free account to see all 76 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have proposed increasingly sophisticated fusion architectures, including attention-based models, variational autoencoders, and graph neural networks, often reporting improved predictive performance in complex biomedical and neuroimaging tasks [6,9,45,46,47].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [9]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In contrast, late fusion offers practical robustness for incomplete or asynchronous clinical data through modular development, albeit without modeling inter-modal relationships”
From Multimodal neuroimaging and AI integration in cognitive disorders: advances, challenges, and future directions for precision medicine (2026) · cites this paper as Huang et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Joint fusion learns a shared embedding space across modalities, which can represent cross-modal relationships beyond those considered in early or late fusion.”
From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A meta-analysis of fusion strategies for combining imaging and EHR data show that multimodal integration leads to better predictive performance in various clinical tasks41.”
From K-STAMM: a knowledge-enhanced spatial - temporal attention model with multimodal fusion for pneumonia prediction (2026) · cites this paper as [41]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Training a model on multiple data types may improve the performance of ML systems (14).”
From Mitigating Bias in Radiology Machine Learning: 1. Data Handling (2022) · cites this paper as [14]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Regularization and Variable Selection Via the Elastic Net
Zou, H., & Hastie, T. (2005) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2005.00503.x · Cited by 20,957
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Early applications of CNNs for image analysis in medicine include diabetic retinopathy, skin cancer, and chest X-rays 11 – 18 .”
BackgroundLarge-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
(2011) · Cited by 2,752
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816
Self-citationRajpurkar, P., Irvin, J., Ball, R. L., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C. P., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002686 · Cited by 1,449 · cited 3×
“The imaging fi ndings on chest X-rays consistent with pneumonia, despite having imaging features that can generally differentiate alternative diagnoses, are nonspeci fi c and accurate diagnosis requires the context of clinical and laboratory data.”
Self-citation
Research works printed but not linked: 2
- Liu, Mingqian, Lan, Jun, Chen, Xu, Yu, Guangjun & Yang, Xiujun Bone age assessment model based on multi-dimensional feature fusion using deep learning. Acad. J. Second Mil. Med. Univ. 39, 909–916 (2018).
- Goodfellow, I., Bengio, Y. & Courville, C. Deep Learning (MIT Press, 2017).
Cited by
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
“Some deep learning models integrate multiple sources of medical data for a multimodal approach 64–68 .”
BackgroundAcosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-022-01981-2 · Cited by 1,365
The Role of AI in Hospitals and Clinics: Transforming Healthcare in the 21st Century
Maleki Varnosfaderani, S., & Forouzanfar, M. (2024) · Bioengineering · doi:10.3390/bioengineering11040337 · Cited by 931
“In the case of complex diseases, AI can combine information from imaging, laboratory tests, and patient histories to provide a more comprehensive diagnostic insight [109].”
BackgroundArtificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777
Multimodal machine learning in precision health: A scoping review
Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., & Luo, Y. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00712-8 · Cited by 528
Metadata
- Authors
- Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P.
- Affiliations
- Stanford University (US)
- Stanford Medicine (US)
- Emory University (US)
- Corresponding author
- Huang, Shih-Cheng
- Venue
- npj Digital Medicine
- Published
- 2020-10-16
- Volume
- 3
- Issue
- 1
- Pages
- 136-136
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC7567861
- OpenAlex
- W3094595351
- Cited by
- 934
- References
- 60
- Citations by year
- 2026: 250 · 2025: 216 · 2024: 187 · 2023: 154 · 2022: 74 · 2021: 49 · 2020: 4
- FWCI
- 2.41
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institutes of Health
- U.S. National Library of Medicine (R01LM012966)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal fusion, diagnostic imaging, electronic health records, deep learning, clinical context, radiology, implementation guidelines
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. https://app.cochat.ai/papers/10.1038/s41746-020-00341-z
- The paper
- Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020). Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. npj Digital Medicine. https://doi.org/10.1038/s41746-020-00341-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.