CoChat

Deep learned tissue “fingerprints” classify breast cancers by ER/PR/Her2 status from H&E images

Open access

The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.

Abstract

Because histologic types are subjective and difficult to reproduce between pathologists, tissue morphology often takes a back seat to molecular testing for the selection of breast cancer treatments. This work explores whether a deep-learning algorithm can learn objective histologic H&E features that predict the clinical subtypes of breast cancer, as assessed by immunostaining for estrogen, progesterone, and Her2 receptors (ER/PR/Her2). Translating deep learning to this and related problems in histopathology presents a challenge due to the lack of large, well-annotated data sets, which are typically required for the algorithms to learn statistically significant discriminatory patterns. To overcome this limitation, we introduce the concept of “tissue fingerprints,” which leverages large, unannotated datasets in a label-free manner to learn H&E features that can distinguish one patient from another. The hypothesis is that training the algorithm to learn the morphological differences between patients will implicitly teach it about the biologic variation between them. Following this training internship, we used the features the network learned, which we call “fingerprints,” to predict ER, PR, and Her2 status in two datasets. Despite the discovery dataset being relatively small by the standards of the machine learning community (n = 939), fingerprints enabled the determination of ER, PR, and Her2 status from whole slide H&E images with 0.89 AUC (ER), 0.81 AUC (PR), and 0.79 AUC (Her2) on a large, independent test set (n = 2531). Tissue fingerprints are concise but meaningful histopathologic image representations that capture biological information and may enable machine learning algorithms that go beyond the traditional ER/PR/Her2 clinical groupings by directly predicting theragnosis.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-020-64156-4

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Networks are first trained to learn tissue fingerprints, which are patterns of cells and tissue visible on H&E images that can be used to distinguish between patients. Following this training internship, which can be scaled to very large numbers of patients without clinical outcome annotations, the fingerprints are repurposed to make clinically relevant predictions from small labeled datasets.
Figure 1 Networks are first trained to learn tissue fingerprints, which are patterns of cells and tissue visible on H&E images that can be used to distinguish between patients. Following this training internship, which can be scaled to very large numbers of patients without clinical outcome annotations, the fingerprints are repurposed to make clinically relevant predictions from small labeled datasets.CC BY · PubMed Central
Figure 2 CycleGAN normalizes the staining colors and styles of tissue images while preserving morphology. Top: the style of slide 1 is translated into the style of slide 2. Bottom: the style of slide 2 is translated into the style of slide 1.
Figure 2 CycleGAN normalizes the staining colors and styles of tissue images while preserving morphology. Top: the style of slide 1 is translated into the style of slide 2. Bottom: the style of slide 2 is translated into the style of slide 1.CC BY · PubMed Central
Figure 3 (a) Representative tSNE visualization of fingerprints from the test set. In this visualization, left halves from slide 5 and right halves of slide 4. (b) Visualization of a representative pair. Left half presented on top, right half on the bottom, middle shows a heat map of fingerprint distance (distance from fingerprints from the bottom image to the average fingerprint of the top image). (c) Left, exploded displays of the original patches in the embedding show similar histologic features (nucleoli, micro-papillae, fat, mucin).
Figure 3 (a) Representative tSNE visualization of fingerprints from the test set. In this visualization, left halves from slide 5 and right halves of slide 4. (b) Visualization of a representative pair. Left half presented on top, right half on the bottom, middle shows a heat map of fingerprint distance (distance from fingerprints from the bottom image to the average fingerprint of the top image). (c) Left, exploded displays of the original patches in the embedding show similar histologic features (nucleoli, micro-papillae, fat, mucin).CC BY · PubMed Central
Figure 4 (a) Heat maps of areas that lead to accurate patient classification. Higher probability regions (red) are more predictive of patient identity, and hence distinctive, than blue regions. (b) An exploded view of two cores from (a).
Figure 4 (a) Heat maps of areas that lead to accurate patient classification. Higher probability regions (red) are more predictive of patient identity, and hence distinctive, than blue regions. (b) An exploded view of two cores from (a).CC BY · PubMed Central
Figure 5 (a) Illustration of whole slide clinical ER classification. An analogous procedure was used for PR and Her2 classification. Fingerprints were extracted from 120 random image patches, and a second ER-classifier, acting on the fingerprints, made local predictions, which were averaged to produce a continuous whole-slide-level ER-score. (b) Receiver operating characteristic curves (ROC) for clinical ER (left), PR (center), and Her2 prediction (right). The TCGA ROC curve reflects a test set from five-fold cross validation, and the AUC corresponds to the average area under the ROC curves of all five TCGA test sets. All samples in the ABCTB dataset are test samples and were never seen during training. Sample sizes vary depending the availability of clinical annotations.
Figure 5 (a) Illustration of whole slide clinical ER classification. An analogous procedure was used for PR and Her2 classification. Fingerprints were extracted from 120 random image patches, and a second ER-classifier, acting on the fingerprints, made local predictions, which were averaged to produce a continuous whole-slide-level ER-score. (b) Receiver operating characteristic curves (ROC) for clinical ER (left), PR (center), and Her2 prediction (right). The TCGA ROC curve reflects a test set from five-fold cross validation, and the AUC corresponds to the average area under the ROC curves of all five TCGA test sets. All samples in the ABCTB dataset are test samples and were never seen during training. Sample sizes vary depending the availability of clinical annotations.CC BY · PubMed Central
Figure 6 (a) Histogram of ER-predictions from the TCGA test set averaged across the entire slide (AUC = 0.88). (b) AUC scores obtained by pooling ER predictions from different regions within slides. (c) Representative heatmaps of correctly and incorrectly classified whole slides. WSI prediction was obtained by averaging over all patches (epithelium, stroma, fat). Each slide visualization consists of an RGB thumbnail, a tissue type segmentation, and an ER prediction heatmap.
Figure 6 (a) Histogram of ER-predictions from the TCGA test set averaged across the entire slide (AUC = 0.88). (b) AUC scores obtained by pooling ER predictions from different regions within slides. (c) Representative heatmaps of correctly and incorrectly classified whole slides. WSI prediction was obtained by averaging over all patches (epithelium, stroma, fat). Each slide visualization consists of an RGB thumbnail, a tissue type segmentation, and an ER prediction heatmap.CC BY · PubMed Central
Figure 7 Left: tSNE embedding of fingerprints from patches extracted from TCGA whole slides, shaded by ER prediction score. 12 clusters with high positive or negative enrichment were selected for manual inspection. Right: H&E patches closest to the cluster centers. Each patch is from a different patient. High resolution image in supplemental information.
Figure 7 Left: tSNE embedding of fingerprints from patches extracted from TCGA whole slides, shaded by ER prediction score. 12 clusters with high positive or negative enrichment were selected for manual inspection. Right: H&E patches closest to the cluster centers. Each patch is from a different patient. High resolution image in supplemental information.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 35 references

Cited by

See all 100

Metadata

Authors
Rawat, R. R., Ortega, I., Roy, P., Sha, F., Shibata, D., Ruderman, D., & Agus, D. B.
Affiliations
University of Southern California (US)
Corresponding author
Ruderman, Daniel
Venue
Scientific Reports
Published
2020-04-29
Volume
10
Issue
1
Pages
7275-7275
ISSN
2045-2322 (ISSN-L)
OpenAlex
W3021500318
Cited by
100
References
35
Citations by year
2026: 13 · 2025: 15 · 2024: 20 · 2023: 18 · 2022: 18 · 2021: 15 · 2020: 1
FWCI
5.11
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • Breast Cancer Research Foundation
  • Oracle
  • National Breast Cancer Foundation
  • Cancer Institute NSW
  • Medical Research Council
  • National Health and Medical Research Council
  • National Cancer Institute (U54CA217376)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Imaging for Blood Diseases
Keywords
breast cancer, H&E histopathology, deep learning, whole slide images, label-free learning, histologic features, molecular subtyping, theragnosis
MeSH terms
Deep Learning, Adult, Aged, Breast Neoplasms, Female, Humans, Image Processing, Computer-Assisted, Middle Aged, Progesterone, Receptors, Estrogen, Biomarkers, Tumor, Receptor, ErbB-2, Erb-b2 Receptor Tyrosine Kinases, Tissue Array Analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Deep learned tissue “fingerprints” classify breast cancers by ER/PR/Her2 status from H&E images. https://app.cochat.ai/papers/10.1038/s41598-020-64156-4
The paper
Rawat, R. R., Ortega, I., Roy, P., Sha, F., Shibata, D., Ruderman, D., & Agus, D. B. (2020). Deep learned tissue “fingerprints” classify breast cancers by ER/PR/Her2 status from H&E images. Scientific Reports. https://doi.org/10.1038/s41598-020-64156-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.