Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 55 of the 60 references it cites were found in a registry or an index; 38 of the 90 sources its claims cite were checked for support.

Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines

Open access

Abstract

In this paper, we describe different data fusion techniques that can be applied to combine medical imaging with EHR, and systematically review medical data fusion literature published between 2012 and 2020. We conducted a systematic search on PubMed and Scopus for original research articles leveraging deep learning for fusion of multimodality data. In total, we screened 985 studies and extracted data from 17 papers. By means of this systematic review, we present current knowledge, summarize important results and provide implementation guidelines to serve as a reference for researchers interested in the application of multimodal data fusion.

Overview

CoChat Summary

TL;DR

This systematic review consolidates deep learning approaches that fuse medical imaging with electronic health record (EHR) data, defining early, joint, and late fusion strategies and reviewing 17 eligible studies published between 2012 and 2020. It concludes that multimodal fusion generally improves performance over single-modality models, though no single fusion strategy is consistently optimal, and provides implementation guidelines for future work.

Key findings

  • From 985 screened studies, 44 underwent full-text screening and 17 met eligibility criteria; early fusion was used in 11/17 studies, joint fusion in 4/17, and late fusion in 3/17.
  • Across reviewed studies, multimodal fusion generally improved performance over single-modality models, with reported accuracy gains of 1.2–27.7% and AUROC gains of 0.02–0.16.
  • No single fusion strategy consistently led to optimal performance across all clinical domains; examples include Kharazmi et al. achieving 91.1% accuracy with fusion versus 84.7% for dermoscopic images and 75.6% for patient profile, and Yala et al. achieving 0.70 AUROC with joint fusion versus 0.68 for mammograms and 0.67 for risk scores.
  • Non-imaging EHR data in the included studies were limited, mostly to basic demographics, a narrow range of categorical clinical history, or disease-specific features such as APOE4 or PSA, rather than feature-rich or time-series EHR data.

Methods

  • PRISMA-based systematic review with searches in PubMed and Scopus for papers published 2012–2020, using key themes of “deep learning,” “multimodality fusion,” and “medical imaging”; segmentation, registration, and reconstruction terms were exclusion criteria, and preprints were excluded.
  • Inclusion required original research applying deep learning for clinical outcome prediction or diagnosis by fusing medical imaging with EHR data; two independent researchers screened records using Covidence, with disagreements resolved by discussion and a third researcher if needed.
  • Data extracted from each included study included fusion strategy, year, authors, clinical domain, target outcome, fusion details, imaging modality, non-imaging modality, number of samples, and model performance; AUROC was extracted when reported, otherwise accuracy, using the best-performing fusion experiment.

Limitations

  • The paper states that its search excluded pre-2012 papers, that publication bias may overrepresent fusion advantages, that it only covered clinical prediction and diagnosis rather than segmentation or registration, and that heterogeneity across studies prevented meta-analysis; it also notes that some studies lacked independent test sets and that classifying unconventional fusion implementations introduced subjectivity.
  • The paper states that the limited number of studies per medical field and heterogeneity made qualitative comparison difficult.
  • Evident from the text: the review includes only 17 studies, and the non-imaging EHR data used across those studies were narrow, limiting conclusions about fusion with richer or time-series clinical data.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-020-00341-z

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Timeline of publications in deep learning for medical imaging. Timeline showing growth in publications on deep learning for medical imaging, found by using the same search criteria on PubMed and Scopus. The figure shows that fusion has only constituted a small, but growing, subset of medical deep learning literature.
Fig. 1 Timeline of publications in deep learning for medical imaging. Timeline showing growth in publications on deep learning for medical imaging, found by using the same search criteria on PubMed and Scopus. The figure shows that fusion has only constituted a small, but growing, subset of medical deep learning literature.CC BY · PubMed Central
Fig. 2 Fusion strategies using deep learning. Model architecture for different fusion strategies. Early fusion (left figure) concatenates original or extracted features at the input level. Joint fusion (middle figure) also joins features at the input level, but the loss is propagated back to the feature extracting model. Late fusion (right figure) aggregates predictions at the decision level.
Fig. 2 Fusion strategies using deep learning. Model architecture for different fusion strategies. Early fusion (left figure) concatenates original or extracted features at the input level. Joint fusion (middle figure) also joins features at the input level, but the loss is propagated back to the feature extracting model. Late fusion (right figure) aggregates predictions at the decision level.CC BY · PubMed Central
Fig. 3 PRISMA flowchart of the study selection process. Two authors independently screened all records for eligibility. Seventeen studies were included in the systematic review.
Fig. 3 PRISMA flowchart of the study selection process. Two authors independently screened all records for eligibility. Seventeen studies were included in the systematic review.CC BY · PubMed Central

Verification

Claims the paper makes

19 Valid · 3 Not valid · 15 Not checked · 39 Own results and interpretations

38 more claims · 9 with their cited sources checked

Create a free account to see all 76 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 20
  • Probably not a substantial claim. 17

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

5 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 60 references

Research works printed but not linked: 2

  1. Liu, Mingqian, Lan, Jun, Chen, Xu, Yu, Guangjun & Yang, Xiujun Bone age assessment model based on multi-dimensional feature fusion using deep learning. Acad. J. Second Mil. Med. Univ. 39, 909–916 (2018).
  2. Goodfellow, I., Bengio, Y. & Courville, C. Deep Learning (MIT Press, 2017).

Cited by

See all 934

Metadata

Authors
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P.
Affiliations
  • Stanford University (US)
  • Stanford Medicine (US)
  • Emory University (US)
Corresponding author
Huang, Shih-Cheng
Venue
npj Digital Medicine
Published
2020-10-16
Volume
3
Issue
1
Pages
136-136
ISSN
2398-6352 (ISSN-L)
OpenAlex
W3094595351
Cited by
934
References
60
Citations by year
2026: 250 · 2025: 216 · 2024: 187 · 2023: 154 · 2022: 74 · 2021: 49 · 2020: 4
FWCI
2.41
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institutes of Health
  • U.S. National Library of Medicine (R01LM012966)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
multimodal fusion, diagnostic imaging, electronic health records, deep learning, clinical context, radiology, implementation guidelines

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. https://app.cochat.ai/papers/10.1038/s41746-020-00341-z
The paper
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020). Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. npj Digital Medicine. https://doi.org/10.1038/s41746-020-00341-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.