Cited by 660AI in cancer detectionRadiomics and Machine Learning in Medical ImagingCancer Genomics and Diagnostics

Pan-cancer integrative histology-genomic analysis via multimodal deep learning

Open access
Chen, Richard J., Lu, Ming Y., Williamson, Drew F.K., Chen, Tiffany Y., Lipkova, Jana, Noor, Zahra, Shaban, Muhammad, Shady, Maha +3 more·2022·Cancer Cell

Abstract

The rapidly emerging field of computational pathology has demonstrated promise in developing objective prognostic models from histology images. However, most prognostic models are either based on histology or genomics alone and do not address how these data sources can be integrated to develop joint image-omic prognostic models. Additionally, identifying explainable morphological and molecular descriptors from these models that govern such prognosis is of interest. We use multimodal deep learning to jointly examine pathology whole-slide images and molecular profile data from 14 cancer types. Our weakly supervised, multimodal deep-learning algorithm is able to fuse these heterogeneous modalities to predict outcomes and discover prognostic features that correlate with poor and favorable outcomes. We present all analyses for morphological and molecular correlates of patient prognosis across the 14 cancer types at both a disease and a patient level in an interactive open-access database to allow for further exploration, biomarker discovery, and feature assessment.

AI snapshot

Written from the paper’s full text.

TL;DR

The authors built a weakly supervised multimodal deep learning model that combines pathology whole-slide images with molecular profiles to predict cancer survival across 14 TCGA cancer types. They report improved risk stratification over unimodal models for most cancers and provide an interactive platform for exploring image and molecular explanations.

Key findings

  • Multimodal fusion achieved an overall cross-validated c-index of 0.645, compared with 0.585 for image-only AMIL and 0.607 for molecular-only SNN; Survival AUC was 0.662 for multimodal, versus 0.616 for SNN and 0.598 for AMIL.
  • The multimodal model had the highest c-index in 12 of 14 cancer types, with statistically significant binary risk stratification in 10 of 14. KIRP showed the largest gain, reaching a c-index of 0.816 versus 0.539 for AMIL and 0.779 for SNN.
  • Whole-slide images contributed an average of 16.8% of input attributions across cancers, but this varied widely, including 55.1% for UCEC, 32.4% for PAAD, and 5.8% for LUSC.
  • Interpretability analyses linked low-risk attention regions to greater immune-cell presence and high-risk regions to tumor presence, higher grade, or invasion; lymphocyte fractions differed significantly in 8 of 14 cancers and tumor-cell fractions in 6 of 14. TIL presence was significantly higher in predicted low-risk patients for 9 of 14 cancers.

Methods

  • The study used 6,592 diagnostic H&E whole-slide images from 5,720 patients across 14 TCGA cancer types, with paired mutation, copy-number, and RNA-Seq data where available.
  • The multimodal model combined an attention-based multiple-instance-learning branch for images, a self-normalizing network for molecular features, and a Kronecker-product fusion layer. Training used patient-level five-fold cross-validation with 80/20 train-test splits, and performance was assessed with c-index, Survival AUC, Kaplan-Meier curves, and logrank tests.
  • Interpretability used attention heatmaps for image regions and Integrated Gradients for molecular features, with HoVerNet-based cell segmentation to quantify cell types and TILs in high-attention regions. Results were made available through the PORPOISE platform.

Limitations

  • The paper states that the approach can identify “what” and “where” but not always “why,” so discovered features require further human quantification and biological interpretation.
  • The paper notes that attention sometimes shifted away from tumor regions toward stroma or normal tissue, and speculates this may reflect training dynamics favoring the simpler modality; molecular data are bulk and lack spatial resolution.
  • Evident from the text: the study is retrospective and TCGA-based, with no independent external validation; SKCM includes metastatic cases that may optimistically bias performance, and data-alignment criteria were relaxed for SKCM and UCEC due to missingness.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.ccell.2022.07.004

Cites

80 of 80 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 2 cited sources checked · 0 supported · 0 flagged

  • Finding

    The paper reports that the multimodal model generally improved performance for nearly all cancer types after integrating modalities.

    “Overall, however, model performances were found to improve following multimodal integration for almost all cancer types”
    Quote in the paperNot yet classified
  • Finding

    Across the 14 cancer types, the multimodal model outperformed the two unimodal baselines on overall c-index and survival AUC.

    “Across the 14 cancer types, MMF achieved an overall c-Index of 0.645, whereas AMIL and SNN had overall c-Indices of 0.585 and 0.607 respectively. On Survival AUC, similar improvement in multimodal integration was found with an overall performance of 0.662 in comparison to 0.616 and 0.598 in SNN and AMIL respectively (Table S2).”
    Quote in the paperNot yet classified
  • Finding

    The multimodal model achieved the highest c-index in 12 of the 14 cancer types.

    “In one-versus-all model performance comparisons on individual cancer types, MMF achieved the highest c-Index on 12 out of 14 (12/14) cancer types, with models for 10/14 cancer types demonstrating statistical significance in binary patient stratification (Figure 2A,2B, 2C).”
    Quote in the paperNot yet classified
  • Finding

    Its multimodal model can fuse the two data types to predict outcomes and identify prognostic features associated with poor and favorable survival.

    “Our weakly-supervised, multimodal deep learning algorithm is able to fuse these heterogeneous modalities to predict outcomes and discover prognostic features that correlate with poor and favorable outcomes.”
    Quote in the paperNot yet classified
  • Finding

    The paper finds that TIL presence is higher in low-risk patients for 9 of the 14 cancer types.

    “we found that 9 out of 14 cancer types had a statistically significant increase in TIL presence amongst patients with predicted low-risk, indicating that model attention was localized to more immune-hot regions.”
    Quote in the paperNot yet classified

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The Chen et al. (2022) model incorporated an explainability component with heatmaps for histopathology and SHAP values for genomic markers.

    “Notably, a vital explainability component was incorporated with heatmaps for histopathology and SHAP values for genomic markers.”

    From Navigating the landscape of multimodal AI in medicine: A scoping review on technical challenges and clinical applications (2025) · cites this paper as Chen et al., 2022

    The Results section, “Multimodal Interpretability for Joint Image-Omic Biomarker Discovery,” describes attention heatmaps over histopathology slides and attribution plots for molecular features (Figures 3–6). It calls the plots SHAP-styled, but says the attribution values were calculated using Integrated Gradients, not SHAP. (LLM, conf=0.88)

    Experimental — accuracy not yet measured
  • In digital pathology applications, whole slide images have been integrated with genomic features for prognosis prediction.

    “and integrated with genomic features for prognosis prediction [20].”
  • In the PORPOISE framework, Chen et al. introduced a method to integrate WSIs with genomic and molecular profiles for prognosis prediction and helped identify poor prognosis-related joint biomarkers.

    “In the PORPOISE framework, Chen et al. introduced a method to integrate WSIs with genomic, and molecular profiles for prognosis prediction and helped identify poor prognosis-related joint biomarkers [20].”
  • Chen et al. (2022) developed a multimodal deep learning model to jointly examine pathology whole-slide images and molecular profile data from 14 cancer types to predict outcomes and discover prognostic features correlating with poor and favorable outcomes.

    “Chen et al. (2022) developed a multimodal deep learning model to jointly examine pathology whole-slide images and molecular profile data from 14 cancer types to predict outcomes and discover prognostic features correlating with poor and favorable outcomes.”
  • The fundamental criterion for evaluating a medical AI agent is its precision in generating diagnostic and treatment recommendations.

    “The fundamental criterion for evaluating a medical AI agent is its precision in generating diagnostic and treatment recommendations.80,100,118,119,120”

    From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [120]

    The Results and Star Methods sections evaluate a cancer-prognosis model using patient risk stratification, c-index, and survival AUC (page not available in our copy). The Introduction and Discussion mention diagnostic tasks and possible future support for treatment decisions, but do not identify the precision of diagnostic and treatment recommendations as the fundamental criterion for evaluating a medical AI agent. Searched for that criterion and for accuracy of diagnostic and treatment recommendations. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • High- and low-risk prognosis labels are reversed

    Minor

    The text assigns high risk to the top quartile and low risk to the bottom quartile, then says those groups have favorable and poor prognoses, respectively. Elsewhere, it associates high risk with poor survival and low risk with favorable survival.

  • PORPOISE has two different expanded names

    Minor

    The platform's name ends in "Integrative Survival Estimation" in the Results but "Integrated Survival Estimation" in Methods.

2 of 2

Cites

Flagged first, then by citations

8 of the 88 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Chen, R. J., Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Noor, Z., Shaban, M., Shady, M., Williams, M., Joo, B., & Mahmood, F.
Affiliations
  • Broad Institute (US)
  • Brigham and Women's Hospital (US)
Corresponding author
Mahmood, Faisal
Venue
Cancer Cell
Publisher
Elsevier BV
Published
2022-08
Volume
40
Issue
8
Pages
865-878.e6
ISSN
1535-6108 (ISSN-L)
OpenAlex
W4291021272
Cited by
660
References
88
Citations by year
2026: 219 · 2025: 215 · 2024: 133 · 2023: 83 · 2022: 9 · 2021: 1
FWCI
36.27
Type
journal article
Language
en
Open access
Locations
  • Cancer Cell (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • U.S. National Library of Medicine (T15LM007092)
  • National Science Foundation
  • Brigham and Women's Hospital
  • National Institutes of Health
  • Google
  • National Human Genome Research Institute
  • National Institute of General Medical Sciences (R35GM138216)
  • National Cancer Institute (T32CA251062)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Cancer Research › Cancer Genomics and Diagnostics
Keywords
pan-cancer, multimodal deep learning, whole slide images, prognostic model, weakly supervised learning, morphological features, molecular profiles, explainable features, biomarker discovery, interactive database
MeSH terms
Deep Learning, Algorithms, Humans, Neoplasms, Prognosis, Genomics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read (extract-v2) on 2026-09-25.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review