Graph attention-based fusion of pathology images and gene expression for prediction of cancer survival

May combine two claims.

The FSM model that leveraged WSI and genomic data shows a marked improvement in performance across both metrics (c-index & tAUC) and datasets (TCGA & CPTAC), outperforming other methods such as Attention MIL [21], Patch-GCN [25], DeepAttnMISL [27], PathomicFusion [28], PORPOISE [5] and MOTCAT [29].

“The FSM model that leveraged WSI and genomic data shows a marked improvement in performance across both metrics (c-index & tAUC) and datasets (TCGA & CPTAC), outperforming other methods such as Attention MIL [21], Patch-GCN [25], DeepAttnMISL [27], PathomicFusion [28], PORPOISE [5] and MOTCAT [29].”

• cites [21] Ilse, Attention-based deep multiple instance learningNot checked yet

  Not checked in the free run.

• cites [25] sourceNot checked yet

  Not checked in the free run.

• cites [27] Yao, 2020, Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks, Med. Image Anal. sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The framework improves survival prediction for human non-small cell lung cancers and outperforms existing state-of-the-art multimodal approaches.

“We show that our framework improves the survival prediction of human non-small cell lung cancers, outperforming existing state-of-the-art approaches that leverage multimodal data.”

• Also stated in: § Contributions · boosted

  “Our experiments show that our framework achieves state-of-the-art performance in predicting survival on human non-small cell lung cancers (NSCLC): lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC), which are the two most common histological types of NSCLCs.”

May combine two claims.

The tAUC values reinforced FSM's robustness, particularly for LUAD cases on the TCGA cohort and LUSC cases on the CPTAC cohort, where FSM's performance was distinctly better (0.679 versus 0.605 and 0.792 versus 0.769, respectively).

“Additionally, the tAUC values reinforced FSM’s robustness, particularly for LUAD cases on the TCGA cohort and LUSC cases on the CPTAC cohort, where FSM’s performance was distinctly better (0.679 versus 0.605 and 0.792 versus 0.769, respectively).”

May combine two claims.

The FSM model achieved higher c-index values for LUAD and LUSC in the TCGA cohort (0.703 versus 0.682 and 0.664 versus 0.640, respectively).

“Specifically, the FSM model achieved higher c-index values for LUAD and LUSC in the TCGA cohort (0.703 versus 0.682 and 0.664 versus 0.640, respectively)”

The quote only partly states this claim.

In the comparison between the FSM and MCAT [6] models across the TCGA and CPTAC cohorts, the FSM model consistently demonstrated superior performance in predicting survival outcomes for both LUAD and LUSC cases.

“In the comparison between the FSM and MCAT [6] models across the TCGA and CPTAC cohorts, the FSM model consistently demonstrated superior performance in predicting survival outcomes for both LUAD and LUSC cases.”

• cites [6] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The model was trained on TCGA whose range of survival time is [4, 7143] in days for LUAD, [0, 4765] in days for LUSC.

“The model was trained on TCGA whose range of survival time is [4, 7143] in days for LUAD, [0, 4765] in days for LUSC.”

May combine two claims.

The LUAD high- and low-risk cases and the LUSC high-risk case all have non-tumor tissue that is not highlighted by the models.

“For example, the LUAD high- and low-risk cases and the LUSC high-risk case all have non-tumor tissue that is not highlighted by the models.”

May combine two claims.

The SAMs for both the ISM and FSM models localized similar neighborhoods as highly associated with patient survival, with FSM model often highlighting additional tumor-specific regions.

“Interestingly, the SAMs for both the ISM and FSM models localized similar neighborhoods as highly associated with patient survival, with FSM model often highlighting additional tumor-specific regions.”

May combine two claims.

The FSM model's SAM, which integrates celltype signatures, aligns closely with pathologist annotations, indicating its clinical utility.

“The FSM model’s SAM, which integrates celltype signatures, aligns closely with pathologist annotations, indicating its clinical utility.”

May combine two claims.

The range of survival time in CPTAC is [0, 1836] days for LUAD, and [0, 1785] days for LUSC.

“The range of survival time in CPTAC is [0, 1836] days for LUAD, and [0, 1785] days for LUSC.”

May combine two claims.

CoAttn visualizations, while occasionally coinciding with pathologic annotations, do not consistently match the SAM's comprehensive coverage, which includes all gene expression signatures for identifying survival-associated regions.

“CoAttn visualizations, while occasionally coinciding with pathologic annotations, do not consistently match the SAM’s comprehensive coverage, which includes all gene expression signatures for identifying survival-associated regions.”

May combine two claims.

ResNet18 is sufficient and more efficient to achieve good performance.

“We observed that ResNet18 is sufficient and more efficient to achieve good performance.”

May combine two claims.

Higher Dice coefficients across various thresholds confirm SAM's enhanced performance, as it captures all gene expression signatures and identifies prognostic areas within the WSI.

“Higher Dice coefficients across various thresholds confirm SAM’s enhanced performance, as it captures all gene expression signatures and identifies prognostic areas within the WSI.”

May combine two claims.

The authors observed a drop in the c-index values on the CPTAC cohort but the tAUC values were relatively similar to the TCGA cohort (Table II).

“We observed a drop in the c-index values on the CPTAC cohort but the tAUC values were relatively similar to the TCGA cohort (Table II).”

May combine two claims.

The model outperforms other SOTA methods using GCN as NML, highlighting the robustness of the approach.

“Experiments show that our model outperforms other SOTA methods using GCN as NML, highlighting the robustness of our approach.”

May combine two claims.

The ISM model's performance highlights its efficacy in utilizing only WSIs, showcasing a competitive edge over other published methods that relied on genomic data or WSIs in isolation.

“The ISM model’s performance highlights its efficacy in utilizing only WSIs, showcasing a competitive edge over other published methods that relied on genomic data or WSIs in isolation.”

May combine two claims. The quote only partly states this claim.

The comparison with state-of-the-art methods used the same 5-fold cross-validation framework, identical SSL-based feature extraction, uniform training hyperparameters and loss functions, and the same genomic signatures.

“For consistent comparison, we employed an identical SSL-based feature extraction process for WSIs and maintained uniform training hyperparameters and loss functions for all models. In cases where genomic data integration was required, we utilized the same genomic signatures identified in our models.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The architecture is unique in its graph-based modeling of local and global features as well as interpretation of image-genomic interactions.

“Our architecture is unique in its graph-based modeling of local and global features as well as interpretation of image-genomic interactions.”

• AssumptionRests on an assumption: The three-dimensional organization of cells into tissue microenvironments has a significant impact on disease development, progression, and outcomes.

May combine two claims.

SAM can incorporate gene expression-specific information on WSIs and generate multimodal spatial signatures that may provide insights into tissue features associated with patient survival.

“SAM can incorporate gene expression-specific information on WSIs and generate multimodal spatial signatures that may provide insights into tissue features associated with patient survival.”

• AssumptionRests on an assumption: The three-dimensional organization of cells into tissue microenvironments has a significant impact on disease development, progression, and outcomes.

May combine two claims.

The model was trained on TCGA using 5-fold cross-validation and tested on CPTAC as an independent dataset.

“We used NLST for generating node-level features for graph construction [8], TCGA for training the models using 5-fold cross-validation, and CPTAC as an independent dataset for testing.”

• cites [8] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

GradCAM is tailored for CNNs and concentrates on the final convolutional layer, so its heatmap is inherently of coarse resolution.

“Specifically tailored for convolutional neural networks (CNNs), GradCAM typically concentrates on the final convolutional layer, emphasizing the significant regions of an image that influence class predictions. Nevertheless, the dimensions of this layer mean the derived heatmap is inherently of a coarse resolution.”

• cites [23] Selvaraju, 2016, Grad-CAM: Visual explanations from deep networks via gradient-based localization, arXiv:1610.02391Not checked yet

  Not checked in the free run.

May combine two claims.

The MLP-Mixer works only on a fixed number of tokens and becomes less effective in handling graph-structured data.

“The well-known MLP-Mixer [7] works only on a fixed number of tokens and becomes less effective in handling graph-structured data.”

• cites [7] Tolstikhin, 2021, MLP-Mixer: An all-MLP architecture for vision, arXiv:2105.01601Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The method can be used to generate hypotheses and identify regions of interest within tissues based on large sample sets that can be further characterized using modern spatial technologies.

“The method can be used to generate hypotheses and identify regions of interest within tissues based on large sample sets that can be further characterized using modern spatial technologies.”

• AssumptionRests on an assumption: Spatial omic data are expensive and technically challenging to generate, resulting in a small number of cases profiled that may only capture a portion of the entire WSI.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

TAH is applicable to models employing attention-weighted pooling of instances, while CoAttn suits multimodal models that fuse WSI and genomic features using co-attention.

“TAH is applicable to models employing attention-weighted pooling of instances, while CoAttn suits multimodal models that fuse WSI and genomic features using co-attention.”

• cites [21] Ilse, Attention-based deep multiple instance learningNot checked yet

  Not checked in the free run.

• cites [6] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

The evaluation is based on the most recent TCGA samples and excludes any samples without corresponding genomic data.

“Our evaluation is based on the most recent TCGA samples and excludes any samples without corresponding genomic data.”

May combine two claims.

SAM produces fine-grained localization maps compared to GradCAM because GradCAM depends on pooled feature maps.

“This approach would produce fine-grained localization maps compared to GradCAM. It is because GradCAM depends on the feature maps from layers that have been subjected to pooling, potentially losing detailed spatial information.”

• cites [23] Selvaraju, 2016, Grad-CAM: Visual explanations from deep networks via gradient-based localization, arXiv:1610.02391Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The authors standardized node features for set-based methods like Attention MIL and DeepAttnMISL and used the same graph inputs for Patch-GCN as in the ISM model.

“We standardized the node features for set-based methods like Attention MIL and DeepAttnMISL and used the same graph inputs for Patch-GCN as in our ISM model.”

May combine two claims.

The attention-based mechanism can highlight the important features in each modality, providing valuable insights into the model's rationale.

“Another significant technical advantage is the interpretability of the model’s decision-making process - the attention-based mechanism can highlight the important features in each modality, providing valuable insights into the model’s rationale.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The censoring bias in survival prediction presents a significant challenge to model training, especially as new datasets may exhibit widely varying censoring rates.

“The censoring bias in survival prediction presents a significant challenge to model training, especially as new datasets may exhibit widely varying censoring rates.”

May combine two claims.

Both NML and CML layers are essential to the approach.

“Experiments shed light on the relative importance of NML versus CML, but both layers are essential to our approach.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In the presence of noise or imperfections in an image, an 8-neighbor connectivity can provide better robustness as it incorporates more information from its neighbors.

“In the presence of noise or imperfections in an image, an 8-neighbor connectivity can provide better robustness as it incorporates more information from its neighbors.”

Probably not a substantial claim: may not pass S3 (load-bearing).

If the proportional hazards assumption is violated (i.e., the hazard ratios are not constant over time), the NLL optimization may produce biased estimates.

“If this assumption is violated (i.e., the hazard ratios are not constant over time), the NLL optimization may produce biased estimates.”

• AssumptionThe authors state this as an assumption: The proportional hazards assumption is integral to the NLL likelihood formulation used in the model.

Probably not a substantial claim: may not pass S3 (load-bearing).

Too much censoring can lead to imprecise estimates, affecting the robustness of the optimization.

“Too much censoring can lead to imprecise estimates, affecting the robustness of the optimization.”

May combine two claims.

By the standards of various metrics, the approach displayed superior performance compared with the SOTA approaches, yielding consistent predictions on two different sample sets - TCGA and CPTAC.

“By the standards of various metrics, our approach displayed superior performance compared with the SOTA approaches, yielding consistent predictions on two different sample sets – TCGA and CPTAC.”

May combine two claims.

Large values in TAH can lead to extremely small attention for the other nodes, which may not reflect the actual uncertainty or variability in the data.

“Large values can lead to extremely small attention for the other nodes, which may not reflect the actual uncertainty or variability in the data.”

May combine two claims.

The framework is capable of generating interpretable maps which compare favorably with expert-driven annotations.

“Our framework is also capable of generating interpretable maps which compare favorably with expert-driven annotations.”

Classified as probably not a substantial claim.

The attention mechanism serves to enhance model performance by focusing on the most relevant aspects of each data modality in a context-aware manner.

“The attention mechanism serves to enhance model performance by focusing on the most relevant aspects of each data modality in a context-aware manner.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Censored observations can complicate the estimation process as they provide partial information about the survival time.

“Censored observations can also complicate the estimation process as they provide partial information about the survival time.”

May combine two claims.

In the study, the authors compared model-based saliency maps with expert-driven annotations on a small set of cases and thus their conclusions are limited.

“In our study, we compared model-based saliency maps with expert-driven annotations on a small set of cases and thus our conclusions are limited.”

Classified as probably not a substantial claim.

If the model is highly sensitive to certain time intervals (performing well in those intervals and poorly elsewhere), this discrepancy could occur.

“If the model is highly sensitive to certain time intervals (performing well in those intervals and poorly elsewhere), this discrepancy could occur.”

May combine two claims.

In conclusion, the graph attention-based approach can efficiently process WSI and bulk genomic data and estimate NSCLC survival.

“In conclusion, our graph attention-based approach can efficiently process WSI and bulk genomic data and estimate NSCLC survival.”

• AssumptionThe authors state this as an assumption: The proportional hazards assumption is integral to the NLL likelihood formulation used in the model.

Probably not a substantial claim: may not pass S3 (load-bearing).

The negative log-likelihood (NLL) loss function used in the model has some limitations that include the assumption that the proportional hazards is integral to the likelihood formulation.

“In addition, the negative log-likelihood (NLL) loss function used in our model has some limitations that include the assumption that the proportional hazards is integral to the likelihood formulation.”

• AssumptionThe authors state this as an assumption: The proportional hazards assumption is integral to the NLL likelihood formulation used in the model.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

By zeroing in on the salient data sections in each modality, the framework boosts computational efficiency, reducing the processing load without compromising on the model performance.

“Furthermore, by zeroing in on the salient data sections in each modality, our framework boosts computational efficiency, reducing the processing load without compromising on the model performance.”

Classified as probably not a substantial claim. May combine two claims.

The reason for this disparity could be due to different sensitivities to time: tAUC is explicitly time-dependent and evaluates the model performance at various time points, whereas the c-index provides a general measure of concordance.

“The reason for this disparity could be due to different sensitivities to time: tAUC is explicitly time-dependent and evaluates the model performance at various time points, whereas the c-index provides a general measure of concordance.”