CoChat

Multimodal deep learning for biomedical data fusion: a review

Open access
Stahlschmidt, Sören Richard, Ulfenborg, Benjamin, Synnergren, Jane · 2022 · Briefings in Bioinformatics

The DOI matches its Crossref record; 107 of the 117 references it cites were found in a registry or an index; 60 of the 96 sources its claims cite were checked for support.

Abstract

Biomedical data are becoming increasingly multimodal and thereby capture the underlying complex relationships among biological processes. Deep learning (DL)-based data fusion strategies are a popular approach for modeling these nonlinear relationships. Therefore, we review the current state-of-the-art of such methods and propose a detailed taxonomy that facilitates more informed choices of fusion strategies for biomedical applications, as well as research on novel methods. By doing so, we find that deep fusion strategies often outperform unimodal and shallow approaches. Additionally, the proposed subcategories of fusion strategies show different advantages and drawbacks. The review of current methods has shown that, especially for intermediate fusion strategies, joint representation learning is the preferred approach as it effectively models the complex interactions of different levels of biological organization. Finally, we note that gradual fusion, based on prior biological knowledge or on search strategies, is a promising future research path. Similarly, utilizing transfer learning might overcome sample size limitations of multimodal data sets. As these data sets become increasingly available, multimodal DL approaches present the opportunity to train holistic models that can learn the complex regulatory dynamics behind health and disease.

Overview

CoChat Summary

TL;DR

This review surveys deep learning-based multimodal biomedical data fusion and proposes a taxonomy of early, intermediate, and late fusion strategies. It finds that deep fusion often outperforms unimodal and shallow approaches, with intermediate joint representation learning preferred, while gradual fusion and transfer learning remain underexplored.

Key findings

  • The paper proposes a taxonomy dividing fusion into early fusion (direct modeling, autoencoder subcategories), intermediate fusion (homogeneous/heterogeneous designs; marginal/joint representations), and late fusion (averaging equal/weighted, meta-learning).
  • Deep fusion strategies frequently outperform unimodal and shallow methods; joint intermediate fusion is the preferred approach because it effectively models complex interactions across levels of biological organization.
  • Early fusion is simple but may miss cross-modal relationships apparent only at higher abstraction; intermediate fusion offers flexibility for heterogeneity, missing modalities, and dimensionality imbalance; late fusion combines decisions but cannot learn feature-level cross-modal interactions.
  • Gradual fusion guided by prior biological knowledge or search strategies and transfer learning are promising future paths; reviewed applications included sample sizes as low as 96 patients.

Methods

  • Narrative literature review of deep learning-based data fusion strategies in the biomedical domain, taking a representation learning perspective.
  • Proposes a detailed taxonomy of early, intermediate, and late fusion and their subcategories, exemplified with published biomedical applications.
  • No primary dataset, systematic search protocol, or quantitative meta-analysis is described; synthesis is conceptual and qualitative.

Limitations

  • The paper states the literature is most likely skewed toward reporting positive results, performance of fusion strategies is problem- and data-specific, and more theoretical knowledge is needed to specify when strategies excel.
  • The paper states multimodal DL faces general challenges including data volume, quality, interpretability, temporality, missing entire modalities, and overfitting/generalizability due to small sample sizes and many parameters.
  • Evident from the text: as a narrative review, it does not report a systematic search strategy or quantitative comparison across studies.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/bib/bbab569

Cites

107 of 107 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Development of technologies and multimodal deep learning (DL). ‘Omics’ and ‘multi-omics’ data become increasingly relevant in the scientific literature. To fully utilize the growing number of multimodal data sets, data fusion methods based on DL are evolving into an important approach in the biomedical field. This unprecedented generation of data has been made possible by high-throughtput technologies like microarrays and next-generation sequencing [7]. The development of bulk RNA-seq was followed by several related sequencing technologies, such as single-cell RNA-seq and ATAC-seq [8]. Currently, spatial transcriptomics [9] and single-cell multi-omics [10] are being increasingly used.
Figure 1 Development of technologies and multimodal deep learning (DL). ‘Omics’ and ‘multi-omics’ data become increasingly relevant in the scientific literature. To fully utilize the growing number of multimodal data sets, data fusion methods based on DL are evolving into an important approach in the biomedical field. This unprecedented generation of data has been made possible by high-throughtput technologies like microarrays and next-generation sequencing [7]. The development of bulk RNA-seq was followed by several related sequencing technologies, such as single-cell RNA-seq and ATAC-seq [8]. Currently, spatial transcriptomics [9] and single-cell multi-omics [10] are being increasingly used.CC BY-NC · PubMed Central
Figure 2 DL-based fusion strategies. Layers marked in blue are shared between modalities and learn joint representations. (a) Early fusion strategies take as input a concatenated vector. No marginal representations are learned. (b) Intermediate fusion strategies first learn marginal representations and fuse these later inside the network. This can occur in one layer or gradually. (c) Late fusion strategies combine decisions by sub-models for each modality. Figure adapted from [2].
Figure 2 DL-based fusion strategies. Layers marked in blue are shared between modalities and learn joint representations. (a) Early fusion strategies take as input a concatenated vector. No marginal representations are learned. (b) Intermediate fusion strategies first learn marginal representations and fuse these later inside the network. This can occur in one layer or gradually. (c) Late fusion strategies combine decisions by sub-models for each modality. Figure adapted from [2].CC BY-NC · PubMed Central
Figure 3 Early fusion strategies. (a) Unimodal vector stacking alternatives. dim(M) is the combined dimensionality of the set of modalities M. m is the number of modalities and t the number of steps. (b) Architecture of a regular AE for early fusion with fusion layer marked in blue. (c) Visualization of the assumptions underlying variational AEs.
Figure 3 Early fusion strategies. (a) Unimodal vector stacking alternatives. dim(M) is the combined dimensionality of the set of modalities M. m is the number of modalities and t the number of steps. (b) Architecture of a regular AE for early fusion with fusion layer marked in blue. (c) Visualization of the assumptions underlying variational AEs.CC BY-NC · PubMed Central
Figure 4 Intermediate fusion strategies. (a) Joint intermediate fusion with shared layer in blue. Subsequent to marginal representations, joint representations are learned (top). In marginal intermediate fusion, marginal representations are directly input to the decision function (bottom). (b) Marginal AE where marginal representations are concatenated and input into a decision function. (c) Joint AE in which a joint representation is learned in the shared layer marked in blue.
Figure 4 Intermediate fusion strategies. (a) Joint intermediate fusion with shared layer in blue. Subsequent to marginal representations, joint representations are learned (top). In marginal intermediate fusion, marginal representations are directly input to the decision function (bottom). (b) Marginal AE where marginal representations are concatenated and input into a decision function. (c) Joint AE in which a joint representation is learned in the shared layer marked in blue.CC BY-NC · PubMed Central

Verification

Claims the paper makes

40 Valid · 5 Not valid · 23 Not checked · 87 Own results and interpretations

91 more claims · 26 with their cited sources checked

Create a free account to see all 155 claims and to verify the 4 references still open.

See all claims — free account
  • It may state two separate claims as one. 19
  • Probably not a substantial claim. 43
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 107 references

Research works printed but not linked: 6

  1. Springer Nature. Milestones in Genomic Sequencing
  2. Goodfellow, 2016, Deep Learning
  3. Ballard, 1987, Modular learning in neural networks. In: Proceedings of the Sixth National Conference on Artificial Intelligence, 279
  4. Kingma, 2014, Proceedings of the International Conference on Learning Representations (ICLR)
  5. Alemi, 2017, 5th International Conference on Learning Representations
  6. Sundararajan, 2017, 34th International Conference on Machine Learning, 5109

Cited by

Items 701–749 of 749

Metadata

Authors
Stahlschmidt, S. R., Ulfenborg, B., & Synnergren, J.
Affiliations
University of Skövde (SE)
Corresponding author
Stahlschmidt, Sören Richard
Venue
Briefings in Bioinformatics
Published
2022-01-28
Volume
23
Issue
2
ISSN
  • 1467-5463 (ISSN-L)
  • 1477-4054
OpenAlex
W4206042814
Cited by
766
References
117
Citations by year
2026: 241 · 2025: 245 · 2024: 175 · 2023: 82 · 2022: 23
FWCI
7.34
Type
journal article
Language
en
Open access
Locations
  • Briefings in Bioinformatics (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
Funders
University of Skövde, Sweden (20170302, 20200014)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
multimodal fusion, biomedical data, deep learning, joint representation learning, intermediate fusion, transfer learning, biological knowledge integration
MeSH terms
Deep Learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.