Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 87 of the 87 references it cites were found in a registry or an index; 30 of the 39 sources its claims cite were checked for support.

Predicting the tumor microenvironment composition and immunotherapy response in non-small cell lung cancer from digital histopathology images

Open access

Published version of a preprint

Abstract

HistoTME predicts 29 BostonGene TME signatures from H&E and uses them to predict immunotherapy response in NSCLC patients treated with pembrolizumab.

Overview

CoChat Summary

## TL;DR

HistoTME is a weakly supervised deep learning method that uses H&E whole-slide histopathology images to infer 30 tumor microenvironment (TME) signatures in NSCLC and to predict immunotherapy response. It achieved an average Pearson correlation of 0.50 on independent tumor cohorts and an AUROC of 0.75 for first-line ICI response prediction in an external clinical cohort of 652 patients.

## Key findings

  • The final HistoTME variant (multi-task AB-MIL + UNI) achieved an average Pearson correlation coefficient of 0.50 against ground truth on the CPTAC-NSCLC validation cohort.
  • On an external SUNY cohort with serial IHC, HistoTME-predicted signatures correlated with measured cell abundances at Pearson 0.60 for T cells, 0.48 for B cells, and 0.41 for macrophages; Spearman correlations were 0.60, 0.54, and 0.49, respectively.
  • Unsupervised clustering of predicted signatures separated patients into 2 TME subtypes, described as Immune-Inflamed and Immune-Desert.
  • For immunotherapy response prediction, interaction-based features improved test-set performance to AUROC 0.68 overall and AUROC 0.75 [95% CI: 0.61-0.88] for patients receiving first-line ICI; using TME signatures alone gave AUROC 0.55 [95% CI: 0.41-0.69].
  • In the SUNY ICI-treated cohort, the Immune-Inflamed vs Immune-Desert status was predictive of overall survival for first-line ICI patients (HR = 0.53 [95% CI: 0.36-0.78], log-rank p = 0.0012) and especially for first-line ICI + chemotherapy (HR = 0.39 [95% CI: 0.22-0.68], log-rank p = 0.00067).

## Methods

  • Design: Weakly supervised deep learning model (HistoTME) trained to predict 30 cell type-specific TME signatures from H&E whole-slide images, then used for unsupervised clustering and supervised ICI response prediction.
  • Data/sample: Trained on TCGA-NSCLC (865 patients; 955 WSIs), validated on CPTAC-NSCLC (333 patients; 1501 WSIs), tested on SUNY Upstate NSCLC with serial IHC (82 patients), and evaluated for clinical response on a retrospective SUNY cohort of 652 patients; 230 patients had response labels for the supervised response model.
  • Main analysis: Tile extraction and feature learning with foundation models (CTransPath, RetCCL, UNI) plus AB-MIL; K-means clustering with silhouette-based selection of K = 2; response modeling with engineered pairwise interaction features (1740) selected by random forest and classified with XGBoost, evaluated by AUROC and survival analyses.

## Limitations

  • The paper states that HistoTME cannot predict the spatial distribution of specific cell types or molecules because it is trained weakly supervised on bulk transcriptomics data.
  • The paper states that there was no additional external validation dataset for ICI response beyond the clinical cohort used here.
  • The work was retrospective and limited to NSCLC, so broader validation and extension to other cancer types are still needed.
  • Only 3 TME signatures were validated with IHC because comparable markers were not available for the others.
  • Evident from the text: predictive accuracy was limited for second- and subsequent-line ICI-treated patients, likely because most patients received first-line therapy and because H&E slides may not reflect later treatment-time TME changes.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41698-024-00765-w

Cites

87 of 87 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Overview of pipeline and performance of HistoTME. A, B Each Whole Slide Image is tessellated into smaller tiles and preprocessed by a pretrained digital pathology foundation model to extract meaningful tile embeddings. The tile embeddings generated by the foundation model are then provided as input to an attention-based multiple instance learning (AB-MIL) module followed by a multi-layer perceptron head (MLP), which learns to predict expression levels to 30 tumor microenvironment-related molecular signatures. Overall, to develop HistoTME we experiment with three open-source foundation models—CTransPath, RetCCL, and UNI21,22,26—and two configurations of AB-MIL: single task AB-MIL, where the predictions of each signature are optimized separately, and multi-task AB-MIL, where predictions of functionally related signatures are jointly optimized. The signature prediction performance of each foundation model coupled with each configuration of AB-MIL is shown on held out CPTAC validation data in Supplementary Fig. 3. Overall, the UNI foundation model + multitask AB-MIL produces the most accurate predictions and is hence chosen as the final version of HistoTME. C Pearson correlations between the ground truth expression levels of each patient derived from bulk transcriptomics and predicted expression levels of each patient derived from the final version of HistoTME (UNI+multi-task AB-MIL) on the held out CPTAC validation cohort. D Pearson and Spearman correlations between the cell type abundance of each patient, defined as the number of marker positive cells per mm2 from immunohistochemistry (IHC) slides, and the predicted cell type-specific signature expression levels of each patient derived from final version of HistoTME (UNI+multitask AB-MIL) is shown on the external SUNY Upstate test cohort. Error bars represent the 95% confidence intervals. Cell type abundances were estimated from whole slide immunohistochemistry images using QuPath v0.5.0 cell detection and classification algorithms with default parameter settings. TME tumor microenvironment, LUAD lung adenocarcinoma, LUSC lung squamous cell carcinoma, MLP multilayer perceptron.
Fig. 1 Overview of pipeline and performance of HistoTME. A, B Each Whole Slide Image is tessellated into smaller tiles and preprocessed by a pretrained digital pathology foundation model to extract meaningful tile embeddings. The tile embeddings generated by the foundation model are then provided as input to an attention-based multiple instance learning (AB-MIL) module followed by a multi-layer perceptron head (MLP), which learns to predict expression levels to 30 tumor microenvironment-related molecular signatures. Overall, to develop HistoTME we experiment with three open-source foundation models—CTransPath, RetCCL, and UNI21,22,26—and two configurations of AB-MIL: single task AB-MIL, where the predictions of each signature are optimized separately, and multi-task AB-MIL, where predictions of functionally related signatures are jointly optimized. The signature prediction performance of each foundation model coupled with each configuration of AB-MIL is shown on held out CPTAC validation data in Supplementary Fig. 3. Overall, the UNI foundation model + multitask AB-MIL produces the most accurate predictions and is hence chosen as the final version of HistoTME. C Pearson correlations between the ground truth expression levels of each patient derived from bulk transcriptomics and predicted expression levels of each patient derived from the final version of HistoTME (UNI+multi-task AB-MIL) on the held out CPTAC validation cohort. D Pearson and Spearman correlations between the cell type abundance of each patient, defined as the number of marker positive cells per mm2 from immunohistochemistry (IHC) slides, and the predicted cell type-specific signature expression levels of each patient derived from final version of HistoTME (UNI+multitask AB-MIL) is shown on the external SUNY Upstate test cohort. Error bars represent the 95% confidence intervals. Cell type abundances were estimated from whole slide immunohistochemistry images using QuPath v0.5.0 cell detection and classification algorithms with default parameter settings. TME tumor microenvironment, LUAD lung adenocarcinoma, LUSC lung squamous cell carcinoma, MLP multilayer perceptron.CC BY · PubMed Central
Fig. 2 Overview of the computational pipeline to classify patients into distinct clusters based on their H&E-predicted TME composition. A H&E stained digitized tumor samples from TCGA + CPTAC are processed by HistoTME and subsequently clustered into two clusters based on partition around medoid (PAM) clustering and a Random Forest classification model that is trained on cluster membership data. B 3D PCA plot visualizing the two distinct clusters of TCGA + CPTAC NSCLC patients: Immune Inflamed and Immune Desert, based on their HistoTME-inferred TME profiles. C Heatmap depicting the H&E-predicted TME composition and clinical attributes of NSCLC patients from the SUNY cohort. Patients were classified into Immune Inflamed cluster or Immune Desert cluster using a two class classification model (Random Forest) trained on TCGA + CPTAC data. D Random forrest-derived feature importance rankings of TME signatures driving the distinction between the Immune Inflamed and Immune Desert cluster. E Immunohistochemistry-derived T cell, B cell and Macrophage abundances, defined as number of marker positive cells per mm2, in cases belonging to the immune-inflamed or immune desert cluster. Cell type abundances were quantified from whole slide immunohistochemistry images using QuPath v0.5.0 positive cell detection and quantification pipline. Statistical significance between groups was determined by non-parametric Wilox rank sum test (***p-value < 0.001, ****p-value < 0.0001).
Fig. 2 Overview of the computational pipeline to classify patients into distinct clusters based on their H&E-predicted TME composition. A H&E stained digitized tumor samples from TCGA + CPTAC are processed by HistoTME and subsequently clustered into two clusters based on partition around medoid (PAM) clustering and a Random Forest classification model that is trained on cluster membership data. B 3D PCA plot visualizing the two distinct clusters of TCGA + CPTAC NSCLC patients: Immune Inflamed and Immune Desert, based on their HistoTME-inferred TME profiles. C Heatmap depicting the H&E-predicted TME composition and clinical attributes of NSCLC patients from the SUNY cohort. Patients were classified into Immune Inflamed cluster or Immune Desert cluster using a two class classification model (Random Forest) trained on TCGA + CPTAC data. D Random forrest-derived feature importance rankings of TME signatures driving the distinction between the Immune Inflamed and Immune Desert cluster. E Immunohistochemistry-derived T cell, B cell and Macrophage abundances, defined as number of marker positive cells per mm2, in cases belonging to the immune-inflamed or immune desert cluster. Cell type abundances were quantified from whole slide immunohistochemistry images using QuPath v0.5.0 positive cell detection and quantification pipline. Statistical significance between groups was determined by non-parametric Wilox rank sum test (***p-value < 0.001, ****p-value < 0.0001).CC BY · PubMed Central
Fig. 3 Overview of HistoTME predictions for a lung resection case predicted to be immune inflamed. A, B Low magnification view of a primary NSCLC tumor resection sample and its predicted TME signature profile. C–E Matched whole slide immunohistochemistry images of the same tumor sample dual stained for CD4 (brown)+CD8 (magenta), CD3 (brown)+CD20(magenta), and P40 (brown)+CD163 (magenta) markers respectively. F HistoTME generated attention maps for each attention head. Below each whole slide attention map are 4 high magnification image tiles (50 × 50 µm) randomly sampled from high attention areas. Supplementary Fig. 6 shows another related example along with higher magnification image tiles randomly sampled from high attention areas.
Fig. 3 Overview of HistoTME predictions for a lung resection case predicted to be immune inflamed. A, B Low magnification view of a primary NSCLC tumor resection sample and its predicted TME signature profile. C–E Matched whole slide immunohistochemistry images of the same tumor sample dual stained for CD4 (brown)+CD8 (magenta), CD3 (brown)+CD20(magenta), and P40 (brown)+CD163 (magenta) markers respectively. F HistoTME generated attention maps for each attention head. Below each whole slide attention map are 4 high magnification image tiles (50 × 50 µm) randomly sampled from high attention areas. Supplementary Fig. 6 shows another related example along with higher magnification image tiles randomly sampled from high attention areas.CC BY · PubMed Central
Fig. 4 Overview of HistoTME predictions for a lung resection case predicted to be immune desert. A, B Low magnification view of a metastatic NSCLC tumor resection sample and its predicted TME signature profile. C–E Matched whole slide immunohistochemistry images of the same tumor sample dual stained for CD4 (brown)+CD8 (magenta)markers, CD3 (brown)+CD20(magenta) markers, and P40 (brown)+CD163 (magenta) markers respectively. F HistoTME generated attention maps for each attention head. Below each whole slide attention map are 4 representative high magnification image tiles (50 × 50 µm) sampled from high attention areas. Supplementary Fig. 7 shows another related example along with higher magnification image tiles randomly sampled from high attention areas.
Fig. 4 Overview of HistoTME predictions for a lung resection case predicted to be immune desert. A, B Low magnification view of a metastatic NSCLC tumor resection sample and its predicted TME signature profile. C–E Matched whole slide immunohistochemistry images of the same tumor sample dual stained for CD4 (brown)+CD8 (magenta)markers, CD3 (brown)+CD20(magenta) markers, and P40 (brown)+CD163 (magenta) markers respectively. F HistoTME generated attention maps for each attention head. Below each whole slide attention map are 4 representative high magnification image tiles (50 × 50 µm) sampled from high attention areas. Supplementary Fig. 7 shows another related example along with higher magnification image tiles randomly sampled from high attention areas.CC BY · PubMed Central
Fig. 5 Association between HistoTME-based TME classification and overall survival outcomes of SUNY NSCLC patients treated with first-line anti-PD1/PD-L1 therapy (first-line IO patients). A Kaplan–Meier plot depicting overall survival−defined as time from date of diagnosis to date of death—of patients that received first-line anti-PD1/PD-L1 treatment. B Kaplan–Meier plot depicting overall survival of SUNY patients that received first-line anti-PD1/PD-L1 therapy stratified by PD-L1 IHC expression. C–E Kaplan–Meier plots depicting overall survival of first-line patients in PD-L1 negative (TPS < 1%), PD-L1 low (TPS = 1–49%) and PD-L1 high (≥50%), cases. Significance of survival differences between distinct subgroups of patients was determined by the log-rank test.
Fig. 5 Association between HistoTME-based TME classification and overall survival outcomes of SUNY NSCLC patients treated with first-line anti-PD1/PD-L1 therapy (first-line IO patients). A Kaplan–Meier plot depicting overall survival−defined as time from date of diagnosis to date of death—of patients that received first-line anti-PD1/PD-L1 treatment. B Kaplan–Meier plot depicting overall survival of SUNY patients that received first-line anti-PD1/PD-L1 therapy stratified by PD-L1 IHC expression. C–E Kaplan–Meier plots depicting overall survival of first-line patients in PD-L1 negative (TPS < 1%), PD-L1 low (TPS = 1–49%) and PD-L1 high (≥50%), cases. Significance of survival differences between distinct subgroups of patients was determined by the log-rank test.CC BY · PubMed Central
Fig. 6 Overview of supervised machine learning model development and validation for predicting immune checkpoint inhibitor responses based on HistoTME predictions. A Model development for response prediction: (1) HistoTME predictions are engineered into new features by taking pairwise sums, differences, products, and quotients. (2) random forest feature selection. (3) XGBoost trained for response prediction. B Feature network pairwise interactions of 18 selected features. Arrow endpoints denote the signature subtracted or divided from the signature at the start point. C Test set receiver operating characteristic (ROC) curve of the model trained on engineered features or TME signatures alone. Optimal cut point shown based on the Youden index. Kaplan Meier plot depicting overall survival of the test set stratified by AI response prediction for D all patients that received anti-PD1/PD-L1 treatment and E first-line immunotherapy (IO)-treated patients. F Shapley additive explanation (SHAP) summary plot ordered by SHAP importance.
Fig. 6 Overview of supervised machine learning model development and validation for predicting immune checkpoint inhibitor responses based on HistoTME predictions. A Model development for response prediction: (1) HistoTME predictions are engineered into new features by taking pairwise sums, differences, products, and quotients. (2) random forest feature selection. (3) XGBoost trained for response prediction. B Feature network pairwise interactions of 18 selected features. Arrow endpoints denote the signature subtracted or divided from the signature at the start point. C Test set receiver operating characteristic (ROC) curve of the model trained on engineered features or TME signatures alone. Optimal cut point shown based on the Youden index. Kaplan Meier plot depicting overall survival of the test set stratified by AI response prediction for D all patients that received anti-PD1/PD-L1 treatment and E first-line immunotherapy (IO)-treated patients. F Shapley additive explanation (SHAP) summary plot ordered by SHAP importance.CC BY · PubMed Central

Verification

Claims the paper makes

18 Valid · 3 Not valid · 4 Not checked · 72 Own results and interpretations

53 more claims · 8 with their cited sources checked

Create a free account to see all 97 claims and to verify the 2 references still open.

See all claims — free account
  • Probably not a substantial claim. 17
  • It may state two separate claims as one. 24

Create a free account to see these claims.

Citations

Cites

Show all 84 references

Research works printed but not linked: 2

  1. Ilse, M., Tomczak, J. & Welling, M. In International Conference on Machine Learning 2127–2136 (PMLR, 2024).
  2. Macenko, M. et al. 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro (Boston, MA, 2009).

Cited by

See all 41

Metadata

Authors
Patkar, S., Chen, A., Basnet, A., Bixby, A., Rajendran, R., Chernet, R., Faso, S., Kumar, P. A., Desai, D., El-Zammar, O., Curtiss, C., Carello, S. J., Nasr, M. R., Choyke, P., Harmon, S., Turkbey, B., & Jamaspishvili, T.
Affiliations
  • National Institutes of Health (US)
  • SUNY Upstate Medical University (US)
Corresponding author
  • Patkar, Sushant
  • Jamaspishvili, Tamara
Venue
npj Precision Oncology
Published
2024-12-19
Volume
8
Issue
1
Pages
280-280
ISSN
2397-768X (ISSN-L)
OpenAlex
W4405596115
Cited by
41
References
84
Citations by year
2026: 27 · 2025: 14
FWCI
6.04
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Precision Oncology (publishedVersion, cc-by)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
National Institutes of Health
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Oncology › Cancer Immunotherapy and Biomarkers
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
non-small cell lung cancer, tumor microenvironment, immune checkpoint inhibitors, immunotherapy response prediction, digital histopathology, whole slide images, weakly supervised deep learning, prognostic factors, PD-L1 expression, cancer immunotherapy

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Predicting the tumor microenvironment composition and immunotherapy response in non-small cell lung cancer from digital histopathology images. https://app.cochat.ai/papers/10.1038/s41698-024-00765-w
The paper
Patkar, S., Chen, A., Basnet, A., Bixby, A., Rajendran, R., Chernet, R., Faso, S., Kumar, P. A., Desai, D., El-Zammar, O., Curtiss, C., Carello, S. J., Nasr, M. R., Choyke, P., Harmon, S., Turkbey, B., & Jamaspishvili, T. (2024). Predicting the tumor microenvironment composition and immunotherapy response in non-small cell lung cancer from digital histopathology images. npj Precision Oncology. https://doi.org/10.1038/s41698-024-00765-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.