CoChat
Sign Up
Cited by 729AI in cancer detectionRadiomics and Machine Learning in Medical ImagingMachine Learning in Healthcare

Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis

Open access
Chen, Richard J., Lu, Ming Y., Wang, Jingwen, Williamson, Drew F. K., Rodig, Scott J., Lindeman, Neal I., Mahmood, Faisal·2022·IEEE Transactions on Medical Imaging
DOI 10.1109/tmi.2020.3021387PMC PMC10339462
Open this paper in CoChat — free accountOpen PDF

Abstract

Cancer diagnosis, prognosis, mymargin and therapeutic response predictions are based on morphological information from histology slides and molecular profiles from genomic data. However, most deep learning-based objective outcome prediction and grading paradigms are based on histology or genomics alone and do not make use of the complementary information in an intuitive manner. In this work, we propose Pathomic Fusion, an interpretable strategy for end-to-end multimodal fusion of histology image and genomic (mutations, CNV, RNA-Seq) features for survival outcome prediction. Our approach models pairwise feature interactions across modalities by taking the Kronecker product of unimodal feature representations, and controls the expressiveness of each representation via a gating-based attention mechanism. Following supervised learning, we are able to interpret and saliently localize features across each modality, and understand how feature importance shifts when conditioning on multimodal input. We validate our approach using glioma and clear cell renal cell carcinoma datasets from the Cancer Genome Atlas (TCGA), which contains paired whole-slide image, genotype, and transcriptome data with ground truth survival and histologic grade labels. In a 15-fold cross-validation, our results demonstrate that the proposed multimodal fusion paradigm improves prognostic determinations from ground truth grading and molecular subtyping, as well as unimodal deep networks trained on histology and genomic data alone. The proposed method establishes insight and theory on how to train deep networks on multimodal biomedical data in an intuitive manner, which will be useful for other problems in medicine that seek to combine heterogeneous data streams for understanding diseases and predicting response and resistance to treatment. Code and trained models are made available at: https://github.com/mahmoodlab/PathomicFusion.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tmi.2020.3021387

Cites

72 of 72 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Multimodal contrastive frameworks have been applied to genotype-imaging integration for neurodegenerative disease, aligning SNP-derived representations with MRI-derived representations to support both predictive accuracy and mechanistic interpretation in Alzheimer's disease diagnosis.

    “Beyond oncology-style histology–omics pairing, multimodal contrastive frameworks have also been applied to genotype–imaging integration for neurodegenerative disease, aligning SNP-derived representations with MRI-derived representations to support both predictive accuracy and mechanistic interpretation in Alzheimer’s disease diagnosis (15).”

    From The role of multimodality in clinical disease diagnosis: advances, challenges, and opportunities (2026) · cites this paper as [15]

    Split into 2 atoms — 0 confirmed, 0 partially supported, 1 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Attention-based fusion methods can weight different latent features by their importance.

    “In addition to element-wise aggregation, attention-based fusion methods can be applied in order to weight different latent features by their importance [74, 79, 81].”

    From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [79]

  • Chen et al. enabled modality-specific interpretability using Grad-CAM and integrated gradient.

    “Chen et al. [79] enabled modality-specific interpretability by applying Grad-CAM [97] for WSI and integrated gradient [98] for cell graph and genomic modalities using convolutional, graph convolutional [14] and fully connected branches.”

    From Multimodal deep learning for biomedical data fusion: a review (2022) · cites this paper as [79]

  • Multimodal models that assume complete data often suffer a significant performance drop when applied in real-world settings.

    “As a result, multimodal models that assume complete data often suffer a significant performance drop when applied in real-world settings.”

    From The role of multimodality in clinical disease diagnosis: advances, challenges, and opportunities (2026) · cites this paper as Chen et al. (17)

    The Experimental Setup > Data Description section reports that approximately 40% of patients in one cohort lacked RNA-Seq expression data. Related Work mentions a different method described as resilient to missing data, but neither passage says that models assuming complete data suffer a significant performance drop in real-world settings. Searches for missing modalities, performance degradation, and real-world deployment found no such finding in our copy; page numbers were unavailable. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

  • Histopathological images and EHRs are two modes of patient data successfully integrated using deep graph learning.

    “Two modes of patient data successfully integrated using deep graph learning are histopathological images [8, 185] and EHRs [186, 187].”

    From Graph representation learning in biomedicine and healthcare (2022) · cites this paper as [8]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 10

  1. [14]Yener B, “ Cell-graphs: image-driven modeling of structure-function relationship,” Communications of the ACM, vol. 60, no. 1, pp. 74– 84, 2016.
  2. [21]Ngiam J, Khosla A, Kim M, Nam J, Lee H, and Ng AY, “ Multimodal deep learning,” in Proceedings of the 28th international conference on machine learning (ICML-11), 2011, pp. 689– 696.
  3. [22]Suresh H, Hunt N, Johnson A, Celi LA, Szolovits P, and Ghassemi M, “ Clinical intervention prediction and understanding with deep neural networks,” in Machine Learning for Healthcare Conference, 2017, pp. 322– 337.
  4. [33]Prewitt JM, “ Graphs and grammars for histology: An introduction,” in Proceedings of the Annual Symposium on Computer Application in Medical Care. American Medical Informatics Association, 1979, p. 18.
  5. [35]Hamilton W, Ying Z, and Leskovec J, “ Inductive representation learning on large graphs,” in Advances in Neural Information Processing Systems, 2017, pp. 1024– 1034.
  6. [37]Defferrard M, Bresson X, and Vandergheynst P, “ Convolutional neural networks on graphs with fast localized spectral filtering,” in Advances in neural information processing systems, 2016, pp. 3844– 3852.
  7. [52]Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, and Bengio Y, “ Generative adversarial nets,” in Advances in neural information processing systems, 2014, pp. 2672– 2680.
  8. [54]Muja M and Lowe DG, “ Fast approximate nearest neighbors with automatic algorithm configuration.” VISAPP (1), vol. 2, no. 331-340, p. 2, 2009.
  9. [58]Klambauer G, Unterthiner T, Mayr A, and Hochreiter S, “ Self-normalizing neural networks,” in Advances in neural information processing systems, 2017, pp. 971– 980.
  10. [62]Sundararajan M, Taly A, and Yan Q, “ Axiomatic attribution for deep networks,” in Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017, pp. 3319– 3328.

Cited by

Sorted by citations

Authors
Chen, R. J., Lu, M. Y., Wang, J., Williamson, D. F. K., Rodig, S. J., Lindeman, N. I., & Mahmood, F.
Affiliations
Brigham and Women's Hospital (US)
ORCID
Venue
IEEE Transactions on Medical Imaging
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2022-04
Volume
41
Issue
4
Pages
757-770
ISSN
  • 0278-0062 (ISSN-L)
  • 1558-254X
DOI
10.1109/tmi.2020.3021387
PMC
PMC10339462
OpenAlex
W2996092187
Cited by
729
References
72
Citations by year
2026: 177 · 2025: 183 · 2024: 155 · 2023: 106 · 2022: 67 · 2021: 37 · 2020: 4
FWCI
21.02
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Medical Imaging (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • Nvidia GPU Grant Program
  • Google Cloud
  • National Science Foundation Graduate Fellowship
  • NIH NHGRI T32 Training Program
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
multimodal fusion, histopathological images, genomic features, survival prediction, cancer prognosis, attention mechanism, Kronecker product, interpretability, feature localization, glioma, clear cell renal cell carcinoma, whole slide images, RNA-seq, copy number variation, genetic mutations, histologic grade
MeSH terms
Glioma, Histological Techniques, Humans, Prognosis, Genomics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review