The DOI matches its Crossref record; 80 of the 88 references it cites were found in a registry or an index; none of the 2 sources its claims cite is checked for support yet.
Pan-cancer integrative histology-genomic analysis via multimodal deep learning
Open accessAbstract
The rapidly emerging field of computational pathology has demonstrated promise in developing objective prognostic models from histology images. However, most prognostic models are either based on histology or genomics alone and do not address how these data sources can be integrated to develop joint image-omic prognostic models. Additionally, identifying explainable morphological and molecular descriptors from these models that govern such prognosis is of interest. We use multimodal deep learning to jointly examine pathology whole-slide images and molecular profile data from 14 cancer types. Our weakly supervised, multimodal deep-learning algorithm is able to fuse these heterogeneous modalities to predict outcomes and discover prognostic features that correlate with poor and favorable outcomes. We present all analyses for morphological and molecular correlates of patient prognosis across the 14 cancer types at both a disease and a patient level in an interactive open-access database to allow for further exploration, biomarker discovery, and feature assessment.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The authors built a weakly supervised multimodal deep learning model that combines pathology whole-slide images with molecular profiles to predict cancer survival across 14 TCGA cancer types. They report improved risk stratification over unimodal models for most cancers and provide an interactive platform for exploring image and molecular explanations.
Key findings
- Multimodal fusion achieved an overall cross-validated c-index of 0.645, compared with 0.585 for image-only AMIL and 0.607 for molecular-only SNN; Survival AUC was 0.662 for multimodal, versus 0.616 for SNN and 0.598 for AMIL.
- The multimodal model had the highest c-index in 12 of 14 cancer types, with statistically significant binary risk stratification in 10 of 14. KIRP showed the largest gain, reaching a c-index of 0.816 versus 0.539 for AMIL and 0.779 for SNN.
- Whole-slide images contributed an average of 16.8% of input attributions across cancers, but this varied widely, including 55.1% for UCEC, 32.4% for PAAD, and 5.8% for LUSC.
- Interpretability analyses linked low-risk attention regions to greater immune-cell presence and high-risk regions to tumor presence, higher grade, or invasion; lymphocyte fractions differed significantly in 8 of 14 cancers and tumor-cell fractions in 6 of 14. TIL presence was significantly higher in predicted low-risk patients for 9 of 14 cancers.
Methods
- The study used 6,592 diagnostic H&E whole-slide images from 5,720 patients across 14 TCGA cancer types, with paired mutation, copy-number, and RNA-Seq data where available.
- The multimodal model combined an attention-based multiple-instance-learning branch for images, a self-normalizing network for molecular features, and a Kronecker-product fusion layer. Training used patient-level five-fold cross-validation with 80/20 train-test splits, and performance was assessed with c-index, Survival AUC, Kaplan-Meier curves, and logrank tests.
- Interpretability used attention heatmaps for image regions and Integrated Gradients for molecular features, with HoVerNet-based cell segmentation to quantify cell types and TILs in high-attention regions. Results were made available through the PORPOISE platform.
Limitations
- The paper states that the approach can identify “what” and “where” but not always “why,” so discovered features require further human quantification and biological interpretation.
- The paper notes that attention sometimes shifted away from tumor regions toward stroma or normal tissue, and speculates this may reflect training dynamics favoring the simpler modality; molecular data are bulk and lack spatial resolution.
- Evident from the text: the study is retrospective and TCGA-based, with no independent external validation; SKCM includes metastatic cases that may optimistically bias performance, and data-alignment criteria were relaxed for SKCM and UCEC due to missingness.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 2 cited sources checked · 0 supported · 0 flagged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, however, model performances were found to improve following multimodal integration for almost all cancer types”
§ Pan-Cancer Integrative Histology-Genomic Analysis via Multimodal Deep Learning > Results > Multimodal Integration Improves Patient Risk Stratification
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across the 14 cancer types, MMF achieved an overall c-Index of 0.645, whereas AMIL and SNN had overall c-Indices of 0.585 and 0.607 respectively. On Survival AUC, similar improvement in multimodal integration was found with an overall performance of 0.662 in comparison to 0.616 and 0.598 in SNN and AMIL respectively (Table S2).”
§ Pan-Cancer Integrative Histology-Genomic Analysis via Multimodal Deep Learning > Results > Multimodal Integration Improves Patient Risk Stratification
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In one-versus-all model performance comparisons on individual cancer types, MMF achieved the highest c-Index on 12 out of 14 (12/14) cancer types, with models for 10/14 cancer types demonstrating statistical significance in binary patient stratification (Figure 2A,2B, 2C).”
§ Pan-Cancer Integrative Histology-Genomic Analysis via Multimodal Deep Learning > Results > Multimodal Integration Improves Patient Risk Stratification
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our weakly-supervised, multimodal deep learning algorithm is able to fuse these heterogeneous modalities to predict outcomes and discover prognostic features that correlate with poor and favorable outcomes.”
§ Pan-Cancer Integrative Histology-Genomic Analysis via Multimodal Deep Learning > Abstract
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we found that 9 out of 14 cancer types had a statistically significant increase in TIL presence amongst patients with predicted low-risk, indicating that model attention was localized to more immune-hot regions.”
§ Pan-Cancer Integrative Histology-Genomic Analysis via Multimodal Deep Learning > Results > Immune response as a prognostic marker
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we note that for MMF models evaluated on UCEC, WSIs contributed to 55.1% of all input attributions, which corroborates with high AMIL performance on this cancer type.”
§ Pan-Cancer Integrative Histology-Genomic Analysis via Multimodal Deep Learning > Results > Multimodal Integration Improves Patient Risk Stratification
In the paper
“Across all cancer types, gradient-based interpretability was able to identify many well-known oncogenes and immune-related genes established in existing biomedical literature and used in targeted molecular therapies (Uhlen et al.,2017).”
§ Pan-Cancer Integrative Histology-Genomic Analysis via Multimodal Deep Learning > Results > Multimodal Interpretability for Joint Image-Omic Biomarker Discovery
What the sources say
- Not checked — no full text of the sourcecites Uhlen et al.,2017 source
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For cancer types in our dataset that have been FDA approved immune checkpoint inhibitor therapies, TIL presence was used as model explanations for favorable prognosis in BRCA (P-Value, 5.17×10−11, t-test), HNSC (P-Value, 1.97×10−18, t-test), KIRC (P-Value, 1.86×10−3, t-test), LIHC (P-Value, 2.54 × 10−17, t-test), LUAD (P-Value, 1.54 × 10−21, t-test), LUSC (P-Value, 2.92 × 10−12, t-test), PAAD (P-Value, 3.77×10−6, t-test), STAD (P-Value, 1.09×10−9, t-test), and SKCM (P-Value, 6.29×10−10, t-test).”
§ Pan-Cancer Integrative Histology-Genomic Analysis via Multimodal Deep Learning > Results > Immune response as a prognostic marker
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“WSIs on average accounted for 16.8% of input attributions in MMF for all cancer types, which suggests that molecular features drive most of the risk prediction in MMF (Figure 2C, Table S3).”
§ Pan-Cancer Integrative Histology-Genomic Analysis via Multimodal Deep Learning > Results > Multimodal Integration Improves Patient Risk Stratification
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all cancer types, we generally observed that high attention regions in low-risk patients corresponded with greater immune cell presence and lower tumor grade than that of high-risk patients, with 8/14 cancer types demonstrating statistically significant differences in lymphocyte cell fractions in high attention regions (Figure 6F, S3-10).”
§ Pan-Cancer Integrative Histology-Genomic Analysis via Multimodal Deep Learning > Results > Multimodal Interpretability for Joint Image-Omic Biomarker Discovery
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Notably, a vital explainability component was incorporated with heatmaps for histopathology and SHAP values for genomic markers.”
From Navigating the landscape of multimodal AI in medicine: A scoping review on technical challenges and clinical applications (2025) · cites this paper as Chen et al., 2022
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“and integrated with genomic features for prognosis prediction [20].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [20]
The cited source says what the claim attributes to it.
In the citing paper
“In the PORPOISE framework, Chen et al. introduced a method to integrate WSIs with genomic, and molecular profiles for prognosis prediction and helped identify poor prognosis-related joint biomarkers [20].”
From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [20]
The cited source says what the claim attributes to it.
In the citing paper
“Chen et al. (2022) developed a multimodal deep learning model to jointly examine pathology whole-slide images and molecular profile data from 14 cancer types to predict outcomes and discover prognostic features correlating with poor and favorable outcomes.”
From Navigating the landscape of multimodal AI in medicine: A scoping review on technical challenges and clinical applications (2025) · cites this paper as Chen et al., 2022
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The fundamental criterion for evaluating a medical AI agent is its precision in generating diagnostic and treatment recommendations.80,100,118,119,120”
From A foundational architecture for AI agents in healthcare (2025) · cites this paper as [120]
Experimental — accuracy not yet measured
In the citing paper
“Additionally, ML and AI allow for integration of multiple data types, including EHRs, genomic data, and environmental exposures, and of multimodal data, such as kinds of information from individual’s various types of records, thus providing a comprehensive perspective of the determinants of health outcomes across the different stages of life course [32].”
From Integrating machine learning and artificial intelligence in life-course epidemiology: pathways to innovative public health solutions (2024) · cites this paper as [32]
In the citing paper
“Multimodal AI systems have been evaluated for various applications in precision oncology, such as outcome predictions14,15.”
From Large language models and multimodal foundation models for precision oncology (2024) · cites this paper as [14]
In the citing paper
“there are increasing efforts of using digital pathology images to guide treatment selection by predicting patient prognosis (73)(74)(75) and/or existing treatment biomarkers (76-79) using AI.”
From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [75]
In the citing paper
“For instance, Chen and colleagues (75) developed an approach to fuse genomic data and histopathology images to predict survival in a pan-cancer cohort.”
From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [75]
In the citing paper
“Recently, several such AI systems have been introduced6. Examples include models that analyze radiology images together with clinical data7 or integrate information from histopathology with genomic8 or text-based information9.”
From Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology (2025) · cites this paper as [8]
In the citing paper
“However, learning the spatial relevance of non-imaging data such as bulk gene expression is not straightforward using late fusion.”
From Graph attention-based fusion of pathology images and gene expression for prediction of cancer survival (2024) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
High- and low-risk prognosis labels are reversed
MinorThe text assigns high risk to the top quartile and low risk to the bottom quartile, then says those groups have favorable and poor prognoses, respectively. Elsewhere, it associates high risk with poor survival and low risk with favorable survival.
“the top 25% (high-risk group) and bottom 25% (low-risk group) of predicted patient risks for each cancer type, which reflect favorable and poor cancer prognosis respectively.”
“WSIs with predicted high-risk scores would have high attention on high tumor grade or tumor invasion regions used in explaining poor survival, whereas WSIs with predicted low risk scores would have high attention on low tumor grade or lymphocyte-containing regions used in explaining favorable survival.”
PORPOISE has two different expanded names
MinorThe platform's name ends in "Integrative Survival Estimation" in the Results but "Integrated Survival Estimation" in Methods.
“Pathology-Omics Research Platform for Integrative Survival Estimation (PORPOISE)”
“PORPOISE (Pathology-Omics Research Platform for Integrated Survival Estimation)”
Citations
Cites
Subramanian, A., Tamayo, P., Mootha, V. K., Mukherjee, S., Ebert, B. L., Gillette, M. A., Paulovich, A., Pomeroy, S. L., Golub, T. R., Lander, E. S., et al. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0506580102 · Cited by 53,625
PD-1 and PD-L1 Immune Checkpoint Blockade to Treat Breast Cancer
Hartkopf, A. D., Taran, F., Wallwiener, M., Walter, C. B., Krämer, B., Grischke, E., & Brucker, S. Y. (2016) · Breast Care · doi:10.1159/000453569 · Cited by 30,989
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989
The 2016 World Health Organization Classification of Tumors of the Central Nervous System: a summary
Louis, D. N., Perry, A., Reifenberger, G., von Deimling, A., Figarella‐Branger, D., Cavenee, W. K., Ohgaki, H., Wiestler, O. D., Kleihues, P., & Ellison, D. W. (2016) · Acta Neuropathologica · doi:10.1007/s00401-016-1545-1 · Cited by 15,703
Tissue-based map of the human proteome
Uhlén, M., Fagerberg, L., Hallström, B. M., Lindskog, C., Oksvold, P., Mardinoglu, A., Sivertsson, Å., Kampf, C., Sjöstedt, E., Asplund, A., et al. (2015) · Science · doi:10.1126/science.1260419 · Cited by 15,286
From local explanations to global understanding with explainable AI for trees
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-019-0138-9 · Cited by 10,551
Mutational landscape determines sensitivity to PD-1 blockade in non–small cell lung cancer
Rizvi, N. A., Hellmann, M. D., Snyder, A., Kvistborg, P., Makarov, V., Havel, J. J., Lee, W., Yuan, J., Wong, P., Ho, T. S., et al. (2015) · Science · doi:10.1126/science.aaa1348 · Cited by 7,802
Amin, M. B., Greene, F. L., Edge, S. B., Compton, C. C., Gershenwald, J. E., Brookland, R. K., Meyer, L., Gress, D. M., Byrd, D. R., & Winchester, D. P. (2017) · CA A Cancer Journal for Clinicians · doi:10.3322/caac.21388 · Cited by 6,531
The Immune Landscape of Cancer
Thorsson, V., Gibbs, D. L., Brown, S. D., Wolf, D., Bortone, D. S., Ou Yang, T.-H., Porta-Pardo, E., Gao, G. F., Plaisier, C. L., Eddy, J. A., et al. (2018) · Immunity · doi:10.1016/j.immuni.2018.03.023 · Cited by 5,870
Mutational heterogeneity in cancer and the search for new cancer-associated genes
Lawrence, M. S., Stojanov, P., Polak, P., Kryukov, G. V., Cibulskis, K., Sivachenko, A., Carter, S. L., Stewart, C., Mermel, C. H., Roberts, S. A., et al. (2013) · Nature · doi:10.1038/nature12213 · Cited by 5,461
Flagged first, then by citations
8 of the 88 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Items 101–109 of 109
Gao, P., Xiao, Q., Tan, H. J. J., Song, J., Fu, Y., Xu, J., Zhao, J., Miao, Y., Li, X., Yi, J., et al. (2024) · Cell Reports Medicine · doi:10.1016/j.xcrm.2024.101848 · Cited by 30
Hou, J., Zhang, R., Xie, Y., Li, C., & Qin, W. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02257-y · Cited by 38
Multistain Pretraining for Slide Representation Learning in Pathology
Jaume, G., Vaidya, A., Zhang, A., H. Song, A., J. Chen, R., Sahai, S., Mo, D., Madrigal, E., Phi Le, L., & Mahmood, F. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73414-4_2 · Cited by 27
Self-citationOne label is all you need: Interpretable AI-enhanced histopathology for oncology
Tavolara, T. E., Su, Z., Gurcan, M. N., & Niazi, M. K. K. (2023) · Seminars in Cancer Biology · doi:10.1016/j.semcancer.2023.09.006 · Cited by 28
Azher, Z. L., Suvarna, A., Chen, J.-Q., Zhang, Z., Christensen, B. C., Salas, L. A., Vaickus, L. J., & Levy, J. J. (2023) · BioData Mining · doi:10.1186/s13040-023-00338-w · Cited by 20
Liu, B., Polack, M., Coudray, N., Claudio Quiros, A., Sakellaropoulos, T., Le, H., Karimkhan, A., Crobach, A. S. L. P., van Krieken, J. H. J. M., Yuan, K., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-57541-y · Cited by 25
“Previous DL studies primarily focused on training models to extract features from WSIs under supervision of potentially extensive and time-consuming human-derived annotations on slide or pixel level, i.e. supervised learning 10,12,15,17 .”
Explainable artificial intelligence for multi-modal cancer analysis: from genomics to immunology
Zhang, M., & Yin, L. (2026) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2025.105040 · Cited by 5
Image Analysis: Understanding and Mitigating Batch Effects in Histopathology
Gindra, R. H., Boushehri, S. S., Wagner, S. J., Tran, M., Winter, D. J. E., Schnabel, J. A., Saur, D., Marr, C., & Peng, T. (2025) · Trillium Pathology · doi:10.47184/tp.2025.01.03 · Cited by 2
The role of multimodality in clinical disease diagnosis: advances, challenges, and opportunities
Nawaz, A., Edinat, A., Rana, M. R. R., Ali, T., Mustafa, G., Tahir, S., & Lee, S. W. (2026) · Frontiers in public health · doi:10.3389/fpubh.2026.1788454 · Cited by 3
Metadata
- Authors
- Chen, R. J., Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Noor, Z., Shaban, M., Shady, M., Williams, M., Joo, B., & Mahmood, F.
- Affiliations
- Broad Institute (US)
- Brigham and Women's Hospital (US)
- Corresponding author
- Mahmood, Faisal
- ORCID
- Venue
- Cancer Cell
- Publisher
- Elsevier BV
- Published
- 2022-08
- Volume
- 40
- Issue
- 8
- Pages
- 865-878.e6
- ISSN
- 1535-6108 (ISSN-L)
- PMC
- PMC10397370
- OpenAlex
- W4291021272
- Cited by
- 660
- References
- 88
- Citations by year
- 2026: 219 · 2025: 215 · 2024: 133 · 2023: 83 · 2022: 9
- FWCI
- 36.27
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Cancer Cell (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- U.S. National Library of Medicine (T15LM007092)
- National Science Foundation
- Brigham and Women's Hospital
- National Institutes of Health
- National Human Genome Research Institute
- National Institute of General Medical Sciences (R35GM138216)
- National Cancer Institute (T32CA251062)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Cancer Research › Cancer Genomics and Diagnostics
- Keywords
- pan-cancer, multimodal deep learning, whole slide images, prognostic model, weakly supervised learning, morphological features, molecular profiles, explainable features, biomarker discovery, interactive database
- MeSH terms
- Deep Learning, Algorithms, Humans, Neoplasms, Prognosis, Genomics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-25 (prompt extract-v2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
