CoChat

The DOI matches its Crossref record; 25 of the 26 references it cites were found in a registry or an index.

Determining breast cancer biomarker status and associated morphological features using deep learning

Open access

Abstract

Background: Breast cancer management depends on biomarkers including estrogen receptor, progesterone receptor, and human epidermal growth factor receptor 2 (ER/PR/HER2). Though existing scoring systems are widely used and well-validated, they can involve costly preparation and variable interpretation. Additionally, discordances between histology and expected biomarker findings can prompt repeat testing to address biological, interpretative, or technical reasons for unexpected results. Methods: We developed three independent deep learning systems (DLS) to directly predict ER/PR/HER2 status for both focal tissue regions (patches) and slides using hematoxylin-and-eosin-stained (H&E) images as input. Models were trained and evaluated using pathologist annotated slides from three data sources. Areas under the receiver operator characteristic curve (AUCs) were calculated for test sets at both a patch-level (>135 million patches, 181 slides) and slide-level ( n = 3274 slides, 1249 cases, 37 sites). Interpretability analyses were performed using Testing with Concept Activation Vectors (TCAV), saliency analysis, and pathologist review of clustered patches. Results: The patch-level AUCs are 0.939 (95%CI 0.936–0.941), 0.938 (0.936–0.940), and 0.808 (0.802–0.813) for ER/PR/HER2, respectively. At the slide level, AUCs are 0.86 (95%CI 0.84–0.87), 0.75 (0.73–0.77), and 0.60 (0.56–0.64) for ER/PR/HER2, respectively. Interpretability analyses show known biomarker-histomorphology associations including associations of low-grade and lobular histology with ER/PR positivity, and increased inflammatory infiltrates with triple-negative staining. Conclusions: This study presents rapid breast cancer biomarker estimation from routine H&E slides and builds on prior advances by prioritizing interpretability of computationally learned features in the context of existing pathological knowledge.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s43856-021-00013-3

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Overview of data annotation, the two-stage deep learning system (DLS), and interpretability techniques. Annotation overview: paired H&E and IHC images were used to develop regional biomarker annotations (see Supplementary Fig. 4). Case-level biomarker status labels were obtained from available pathology reports. DLS overview: a model based on the Inception-v3 architecture was developed for each biomarker. Model interpretability: saliency maps and unsupervised clustering provided an exploratory approach to interpretability, while concept activation vector analysis provided hypothesis-driven analysis of features associated with biomarker predictions. H&E Hematoxylin and Eosin, ER Estrogen Receptor, PR Progesterone Receptor, HER2 human epidermal growth factor receptor 2.
Fig. 1 Overview of data annotation, the two-stage deep learning system (DLS), and interpretability techniques. Annotation overview: paired H&E and IHC images were used to develop regional biomarker annotations (see Supplementary Fig. 4). Case-level biomarker status labels were obtained from available pathology reports. DLS overview: a model based on the Inception-v3 architecture was developed for each biomarker. Model interpretability: saliency maps and unsupervised clustering provided an exploratory approach to interpretability, while concept activation vector analysis provided hypothesis-driven analysis of features associated with biomarker predictions. H&E Hematoxylin and Eosin, ER Estrogen Receptor, PR Progesterone Receptor, HER2 human epidermal growth factor receptor 2.CC BY · PubMed Central
Fig. 2 Deep learning system performance. ROC curves for model performance are shown for a patch-level predictions across all tissue patches of WSIs, b slide-level predictions of the stage 2 model output on the full test set, c–d subanalysis for slide-level performance on the independent data sources of the slide level test set. Patch-level analysis represents 3-class performance (biomarker positive invasive carcinoma, biomarker negative invasive carcinoma, or non-tumor) and slide-level performance represents positive versus negative classification for biomarker status (all slides in the final datasets contain tumor). Binary patch-level performance for biomarker status across tumor regions only are shown in Supplementary Fig. 5. The number of slides, cases, and patches used for this analysis are indicated in Tables 1 and 2. ER Estrogen Receptor, PR Progesterone Receptor, HER2 human epidermal growth factor receptor 2.
Fig. 2 Deep learning system performance. ROC curves for model performance are shown for a patch-level predictions across all tissue patches of WSIs, b slide-level predictions of the stage 2 model output on the full test set, c–d subanalysis for slide-level performance on the independent data sources of the slide level test set. Patch-level analysis represents 3-class performance (biomarker positive invasive carcinoma, biomarker negative invasive carcinoma, or non-tumor) and slide-level performance represents positive versus negative classification for biomarker status (all slides in the final datasets contain tumor). Binary patch-level performance for biomarker status across tumor regions only are shown in Supplementary Fig. 5. The number of slides, cases, and patches used for this analysis are indicated in Tables 1 and 2. ER Estrogen Receptor, PR Progesterone Receptor, HER2 human epidermal growth factor receptor 2.CC BY · PubMed Central
Fig. 3 Visualization of predictions and annotations. a Sample cases for which DLS predictions are concordant with region-level pathologist annotations. b Sample cases for which DLS predictions are discordant with region-level pathologist annotations. Predictions are represented by heatmaps with colors ranging from blue (low predicted probability) to red (high predicted probability), and with black indicating non-tissue. In the annotation masks (black and white), annotations are represented by white regions, and corresponding labels indicated in the column header. H&E Hematoxylin and Eosin, IHC Immunohistochemistry, DLS Deep Learning System, ER Estrogen Receptor, PR Progesterone Receptor, HER2 human epidermal growth factor receptor 2.
Fig. 3 Visualization of predictions and annotations. a Sample cases for which DLS predictions are concordant with region-level pathologist annotations. b Sample cases for which DLS predictions are discordant with region-level pathologist annotations. Predictions are represented by heatmaps with colors ranging from blue (low predicted probability) to red (high predicted probability), and with black indicating non-tissue. In the annotation masks (black and white), annotations are represented by white regions, and corresponding labels indicated in the column header. H&E Hematoxylin and Eosin, IHC Immunohistochemistry, DLS Deep Learning System, ER Estrogen Receptor, PR Progesterone Receptor, HER2 human epidermal growth factor receptor 2.CC BY · PubMed Central
Fig. 4 Testing with Concept Activation Vector (TCAV) analysis. a Representative concept patches for each of the six concepts used for TCAV analysis. b TCAV scores for positive (blue) and negative (red) biomarker status for each of the three biomarkers. Error bars are 95% confidence intervals over 20 trials using 500 class-of-interest patches, 500 random patches, and 100 concept patches per trial. Higher scores indicate stronger association of the concept with the model’s representation of that biomarker status. Detailed TCAV scores are provided in Supplementary Table 3 and Supplementary Data 2. TIL tumor infiltrating lymphocytes, ER Estrogen Receptor, PR Progesterone Receptor, HER2 human epidermal growth factor receptor 2, DCIS Ductal Carcinoma In Situ.
Fig. 4 Testing with Concept Activation Vector (TCAV) analysis. a Representative concept patches for each of the six concepts used for TCAV analysis. b TCAV scores for positive (blue) and negative (red) biomarker status for each of the three biomarkers. Error bars are 95% confidence intervals over 20 trials using 500 class-of-interest patches, 500 random patches, and 100 concept patches per trial. Higher scores indicate stronger association of the concept with the model’s representation of that biomarker status. Detailed TCAV scores are provided in Supplementary Table 3 and Supplementary Data 2. TIL tumor infiltrating lymphocytes, ER Estrogen Receptor, PR Progesterone Receptor, HER2 human epidermal growth factor receptor 2, DCIS Ductal Carcinoma In Situ.CC BY · PubMed Central
Fig. 5 Visualizing similar patches grouped by predicted biomarker status. Sample patches from clusters belonging to each biomarker status prediction grouping are shown. First, patches were clustered based on visual similarity (Cluster 0–24), and then clusters were grouped by applying a second clustering step (hierarchical clustering) using the predicted biomarker status distributions for each cluster (as visualized in the dendrogram). Distributions of biomarker status predictions within each cluster and additional patches for each cluster are shown in Supplementary Fig. 2a–d. Pathologist-provided descriptions of tumor and stromal features for each cluster are summarized in Supplementary Data 1 with additional biomarker prediction and cluster data provided as Supplementary Data 3. For the sample clusters with representative patches in this figure, brief descriptions are as follows: clusters 10 and 1: largely low and intermediate grade tumor with high stromal content; clusters 24 and 8: intermediate grade tumor with diffuse pattern of invasion resulting in a predominant adipose tissue component; cluster 14: intermediate grade tumor with inflammatory cell infiltrates; cluster 15: intermediate grade tumor forming sheets and occasional cribriform morphology; cluster 4: intermediate grade tumor forming cords; clusters 20, 17, and 9: low (17), intermediate (9), and high-grade (20) tumor with variable architecture, occasionally cribriform morphology, and stroma with moderate sclerosis; cluster 23 and 22: intermediate grade, high-tumor-content patches with moderate sclerosis; cluster 19 and 2: low tumor content and presence of extracellular mucin; clusters 11 and 13: high-intermediate grade tumor and high tumor content as well as inflammatory cell infiltrates. Images patches all represent 1024 μM × 1024 μM ER Estrogen Receptor, PR Progesterone Receptor.
Fig. 5 Visualizing similar patches grouped by predicted biomarker status. Sample patches from clusters belonging to each biomarker status prediction grouping are shown. First, patches were clustered based on visual similarity (Cluster 0–24), and then clusters were grouped by applying a second clustering step (hierarchical clustering) using the predicted biomarker status distributions for each cluster (as visualized in the dendrogram). Distributions of biomarker status predictions within each cluster and additional patches for each cluster are shown in Supplementary Fig. 2a–d. Pathologist-provided descriptions of tumor and stromal features for each cluster are summarized in Supplementary Data 1 with additional biomarker prediction and cluster data provided as Supplementary Data 3. For the sample clusters with representative patches in this figure, brief descriptions are as follows: clusters 10 and 1: largely low and intermediate grade tumor with high stromal content; clusters 24 and 8: intermediate grade tumor with diffuse pattern of invasion resulting in a predominant adipose tissue component; cluster 14: intermediate grade tumor with inflammatory cell infiltrates; cluster 15: intermediate grade tumor forming sheets and occasional cribriform morphology; cluster 4: intermediate grade tumor forming cords; clusters 20, 17, and 9: low (17), intermediate (9), and high-grade (20) tumor with variable architecture, occasionally cribriform morphology, and stroma with moderate sclerosis; cluster 23 and 22: intermediate grade, high-tumor-content patches with moderate sclerosis; cluster 19 and 2: low tumor content and presence of extracellular mucin; clusters 11 and 13: high-intermediate grade tumor and high tumor content as well as inflammatory cell infiltrates. Images patches all represent 1024 μM × 1024 μM ER Estrogen Receptor, PR Progesterone Receptor.CC BY · PubMed Central
Fig. 6 Saliency analysis for patch-level biomarker status prediction. Examples of saliency maps for patches representing each biomarker status prediction; ER (a), PR (b), and HER2 (c). Inset regions highlight higher magnification on both H&E and IHC for the most relevant regions of saliency analysis. Warm colors on the overlay indicate that the underlying pixels from the H&E slide were salient to the model’s predicted probability for the indicated biomarker status. ER Estrogen Receptor, PR Progesterone Receptor, HER2 human epidermal growth factor receptor 2.
Fig. 6 Saliency analysis for patch-level biomarker status prediction. Examples of saliency maps for patches representing each biomarker status prediction; ER (a), PR (b), and HER2 (c). Inset regions highlight higher magnification on both H&E and IHC for the most relevant regions of saliency analysis. Warm colors on the overlay indicate that the underlying pixels from the H&E slide were salient to the model’s predicted probability for the indicated biomarker status. ER Estrogen Receptor, PR Progesterone Receptor, HER2 human epidermal growth factor receptor 2.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 25 references

Research works printed but not linked: 1

  1. Juan, D.-C. et al. Graph-RISE: Graph-Regularized Image Semantic Embedding. arXiv [cs.CV] (2019).

Cited by

See all 177

Metadata

Authors
Gamble, P., Jaroensri, R., Wang, H., Tan, F., Moran, M., Brown, T., Flament-Auvigne, I., Rakha, E. A., Toss, M., Dabbs, D. J., Regitnig, P., Olson, N., Wren, J. H., Robinson, C., Corrado, G. S., Peng, L. H., Liu, Y., Mermel, C. H., Steiner, D. F., & Chen, P.-H. C.
Affiliations
  • Google (United States) (US)
  • University of Nottingham (GB)
  • UPMC Health System (US)
  • University of Graz (AT)
  • United States Department of Defense (US)
  • Henry M. Jackson Foundation (US)
  • Naval Medical Center San Diego (US)
Corresponding author
  • Steiner, David F.
  • Chen, Po-Hsuan Cameron
Venue
Communications Medicine
Published
2021-07-14
Volume
1
Issue
1
Pages
14-14
ISSN
2730-664X (ISSN-L)
OpenAlex
W3177811843
Cited by
154
References
26
Citations by year
2026: 20 · 2025: 45 · 2024: 32 · 2023: 30 · 2022: 21 · 2021: 6
FWCI
7.83
Type
journal article
Language
en
Open access
Licence
Locations
  • Communications Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • U.S. Department of Defense
  • Henry M. Jackson Foundation
  • Google
  • Verily Life Sciences
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Cancer Research › Breast Cancer Treatment Studies
Keywords
breast cancer, deep learning, H&E-stained slides, histomorphology, model interpretability, saliency analysis, triple-negative breast cancer, inflammatory cell infiltration, lobular histology, histopathology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Determining breast cancer biomarker status and associated morphological features using deep learning. https://app.cochat.ai/papers/10.1038/s43856-021-00013-3
The paper
Gamble, P., Jaroensri, R., Wang, H., Tan, F., Moran, M., Brown, T., Flament-Auvigne, I., Rakha, E. A., Toss, M., Dabbs, D. J., Regitnig, P., Olson, N., Wren, J. H., Robinson, C., Corrado, G. S., Peng, L. H., Liu, Y., Mermel, C. H., Steiner, D. F., & Chen, P.-H. C. (2021). Determining breast cancer biomarker status and associated morphological features using deep learning. Communications Medicine. https://doi.org/10.1038/s43856-021-00013-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.