Multimodal deep learning for cancer prognosis prediction with clinical information prompts integration

Abstract

Survival prediction is crucial for guiding cancer treatment and evaluating therapeutic efficacy. However, tumor heterogeneity presents challenges of accurate prognosis. Multimodal learning, which integrates data from imaging, genomics, and clinical records, offers a promising approach for this complex task. While recent studies mainly focus on imaging and genomic data, clinical information, which reflecting patients' overall health, remains underutilized due to its discrete, sparse, and low-dimensional characteristics. We propose SurvPGC, an integrated model combining pathology images, genomic data and clinical records for cancer prognosis. Clinical information is transformed into high-dimensional vectors using text templates and foundation models, enabling their integration through a cross-attention module. Validation on three datasets of The Cancer Genome Atlas demonstrated that the model effectively captures modality-specific features, with attention visualization revealing distinct focus areas across data types. This highlights the importance of incorporating diverse information sources for improved survival prediction.