Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 87 of the 87 references it cites were found in a registry or an index.

The impact of site-specific digital histology signatures on deep learning model accuracy and bias

Open access

Published version of a preprint

Abstract

The Cancer Genome Atlas (TCGA) is one of the largest biorepositories of digital histology. Deep learning (DL) models have been trained on TCGA to predict numerous features directly from histology, including survival, gene expression patterns, and driver mutations. However, we demonstrate that these features vary substantially across tissue submitting sites in TCGA for over 3,000 patients with six cancer subtypes. Additionally, we show that histologic image differences between submitting sites can easily be identified with DL. Site detection remains possible despite commonly used color normalization and augmentation methods, and we quantify the image characteristics constituting this site-specific digital histology signature. We demonstrate that these site-specific signatures lead to biased accuracy for prediction of features including survival, genomic mutations, and tumor stage. Furthermore, ethnicity can also be inferred from site-specific signatures, which must be accounted for to ensure equitable application of DL. These site-specific signatures can lead to overoptimistic estimates of model performance, and we propose a quadratic programming method that abrogates this bias by ensuring models are not trained and validated on samples from the same site.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41467-021-24698-1

Cites

87 of 87 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Etiologies of site-specific digital histology signatures, and methods for correction. The submitting institution of a digital histology image can often be readily detected due to a site-specific signature unique to each institution. A number of factors can contribute to site-specific signatures, ranging from true histologic and biologic differences between datasets to nonbiologic artifacts, termed batch effect. The batch effect can originate from every step from the procurement of tissue to digital image creation. Frozen and formalin-fixed specimens will have unique histologic artifacts, the intensity of hematoxylin and eosin exposure can vary between institutions, and the digitization of slides may result in compression artifacts. A variety of methods have been developed to mitigate the impact of stain differences between slides. Stain normalization refers to changes in color characteristics to reduce the effect of staining differences between slides. Augmentation refers to random variations applied to individual tiles during machine learning to prevent overfitting with regards to the varied characteristic.
Fig. 1 Etiologies of site-specific digital histology signatures, and methods for correction. The submitting institution of a digital histology image can often be readily detected due to a site-specific signature unique to each institution. A number of factors can contribute to site-specific signatures, ranging from true histologic and biologic differences between datasets to nonbiologic artifacts, termed batch effect. The batch effect can originate from every step from the procurement of tissue to digital image creation. Frozen and formalin-fixed specimens will have unique histologic artifacts, the intensity of hematoxylin and eosin exposure can vary between institutions, and the digitization of slides may result in compression artifacts. A variety of methods have been developed to mitigate the impact of stain differences between slides. Stain normalization refers to changes in color characteristics to reduce the effect of staining differences between slides. Augmentation refers to random variations applied to individual tiles during machine learning to prevent overfitting with regards to the varied characteristic.CC BY · PubMed Central
Fig. 2 Demographics and tumor characteristics of breast cancer across sites with 20 or more slides in TCGA. Each row represents a demographic, clinical, or tumor characteristic of patients in TCGA-BRCA. The chi-squared test was performed to quantify heterogeneity between sites, with the listed P values corrected for a false discovery rate of 0.05. A number of features display marked heterogeneity—for example, only two sites (ILSBio and Christiana) submitted patients where the majority had disease progression within 3 years. IGC International Genomics Consortium, MSKCC Memorial Sloan Kettering Cancer Center, GPCC Greater Poland Cancer Center, EUR European, AFR African, AMR Native American, IDC invasive ductal carcinoma, ILC invasive lobular carcinoma.
Fig. 2 Demographics and tumor characteristics of breast cancer across sites with 20 or more slides in TCGA. Each row represents a demographic, clinical, or tumor characteristic of patients in TCGA-BRCA. The chi-squared test was performed to quantify heterogeneity between sites, with the listed P values corrected for a false discovery rate of 0.05. A number of features display marked heterogeneity—for example, only two sites (ILSBio and Christiana) submitted patients where the majority had disease progression within 3 years. IGC International Genomics Consortium, MSKCC Memorial Sloan Kettering Cancer Center, GPCC Greater Poland Cancer Center, EUR European, AFR African, AMR Native American, IDC invasive ductal carcinoma, ILC invasive lobular carcinoma.CC BY · PubMed Central
Fig. 3 Variation of image characteristics in breast cancer digital histology across TCGA. Sites contributing at least 50 slides are included (n = 607 slides, 7 sites), demonstrating that image variation is not solely a function of small sites that infrequently contributed to TCGA. a First-order characteristics for red, green, and blue are shown in their respective colors. b Haralick second-order textural features also vary by submitting site. STD standard deviation, ASM angular second moment, GPCC Greater Poland Cancer Center.
Fig. 3 Variation of image characteristics in breast cancer digital histology across TCGA. Sites contributing at least 50 slides are included (n = 607 slides, 7 sites), demonstrating that image variation is not solely a function of small sites that infrequently contributed to TCGA. a First-order characteristics for red, green, and blue are shown in their respective colors. b Haralick second-order textural features also vary by submitting site. STD standard deviation, ASM angular second moment, GPCC Greater Poland Cancer Center.CC BY · PubMed Central
Fig. 4 ANOVA F-statistic for first- and second-order image characteristics for breast cancer histology in TCGA. Variance in first- and second-order image characteristics between tissue submitting sites in the breast cancer TCGA dataset (n = 888 slides over 14 sites) is assessed with ANOVA. The ANOVA F-statistic is listed for multiple methods of stain normalization, with the lowest F-statistic (least variability) with any method of normalization indicated in the rightmost column. Stain normalization does not completely resolve first-order stain variability (by F-statistic), and minimal impact is seen on second-order Haralick features. STD standard deviation, ASM angular second moment.
Fig. 4 ANOVA F-statistic for first- and second-order image characteristics for breast cancer histology in TCGA. Variance in first- and second-order image characteristics between tissue submitting sites in the breast cancer TCGA dataset (n = 888 slides over 14 sites) is assessed with ANOVA. The ANOVA F-statistic is listed for multiple methods of stain normalization, with the lowest F-statistic (least variability) with any method of normalization indicated in the rightmost column. Stain normalization does not completely resolve first-order stain variability (by F-statistic), and minimal impact is seen on second-order Haralick features. STD standard deviation, ASM angular second moment.CC BY · PubMed Central
Fig. 5 Model development for the site and feature prediction for patients in TCGA. a To predict tissue submitting site, data is split into threefolds, with each site represented equally in all folds. Cross-validation is then performed, where a model is trained on two of the datasets and performance is assessed on the third dataset. This process is repeated threefold for an averaged performance metric. b UMAP representation of final activation weight vector of the model trained to recognize submitting site in TCGA-BRCA (n = 1006 slides). Each point on the left figure represents the centroid tile from a single slide. The nearest tile to each UMAP coordinate is visualized on the right, demonstrating a clear gradient from tiles that demonstrate predominant hematoxylin staining to those demonstrating predominant eosin. c We assess the impact of including slides from a tissue submitting site within both the training and validation sets on the prediction of a variety of clinical, genomic, and demographic features, using two methods of generating folds for cross-validation. First, we split the data into threefolds, stratifying by the feature of interest, irrespective of site. For a comparator, we split the data into threefolds where each site is isolated into a single fold, with the secondary objective of equalizing the ratio of features in each fold.
Fig. 5 Model development for the site and feature prediction for patients in TCGA. a To predict tissue submitting site, data is split into threefolds, with each site represented equally in all folds. Cross-validation is then performed, where a model is trained on two of the datasets and performance is assessed on the third dataset. This process is repeated threefold for an averaged performance metric. b UMAP representation of final activation weight vector of the model trained to recognize submitting site in TCGA-BRCA (n = 1006 slides). Each point on the left figure represents the centroid tile from a single slide. The nearest tile to each UMAP coordinate is visualized on the right, demonstrating a clear gradient from tiles that demonstrate predominant hematoxylin staining to those demonstrating predominant eosin. c We assess the impact of including slides from a tissue submitting site within both the training and validation sets on the prediction of a variety of clinical, genomic, and demographic features, using two methods of generating folds for cross-validation. First, we split the data into threefolds, stratifying by the feature of interest, irrespective of site. For a comparator, we split the data into threefolds where each site is isolated into a single fold, with the secondary objective of equalizing the ratio of features in each fold.CC BY · PubMed Central
Fig. 6 Impact of site-specific digital histology signatures on deep-learning model accuracy and bias. a The distribution of the average difference in AUROC with standard and preserved-site cross-validation for various clinical, genomic, and demographic features (n = 58 features) for six cancer subtypes in TCGA is shown (pictured graphic for baseline models without normalization/augmentation). The decrease in AUROC is statistically significant for a number of features (one-sided t-test, illustrated on y axis with false discovery correction, as described in Supplementary Table 7) for a subset of features. Jitter is added to ease visualization, although significance/insignificance of individual findings is preserved. b The same graph is provided for grayscale stain adjustment (for which the smallest changes in AUROC were seen). c False-positive prediction of European ancestry and African ancestry for patients within the University of Chicago dataset (measured at the tile level, n = 2206 tiles from 20 patients, 17 with African ancestry, 3 with European ancestry) for models trained with standard and preserved-site cross-validation. The presented bars illustrate the proportion (e.g., the number of tiles falsely predicted to be European divided by the total number of tiles predicted to be European), with error bars signifying the estimated standard deviation of the proportions. PFS progression-free survival, HSV hue saturation value.
Fig. 6 Impact of site-specific digital histology signatures on deep-learning model accuracy and bias. a The distribution of the average difference in AUROC with standard and preserved-site cross-validation for various clinical, genomic, and demographic features (n = 58 features) for six cancer subtypes in TCGA is shown (pictured graphic for baseline models without normalization/augmentation). The decrease in AUROC is statistically significant for a number of features (one-sided t-test, illustrated on y axis with false discovery correction, as described in Supplementary Table 7) for a subset of features. Jitter is added to ease visualization, although significance/insignificance of individual findings is preserved. b The same graph is provided for grayscale stain adjustment (for which the smallest changes in AUROC were seen). c False-positive prediction of European ancestry and African ancestry for patients within the University of Chicago dataset (measured at the tile level, n = 2206 tiles from 20 patients, 17 with African ancestry, 3 with European ancestry) for models trained with standard and preserved-site cross-validation. The presented bars illustrate the proportion (e.g., the number of tiles falsely predicted to be European divided by the total number of tiles predicted to be European), with error bars signifying the estimated standard deviation of the proportions. PFS progression-free survival, HSV hue saturation value.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 83 references

Research works printed but not linked: 1

  1. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2017). · arXiv

Cited by

See all 436

Metadata

Authors
Howard, F. M., Dolezal, J., Kochanny, S., Schulte, J., Chen, H., Heij, L., Huo, D., Nanda, R., Olopade, O. I., Kather, J. N., Cipriani, N., Grossman, R. L., & Pearson, A. T.
Affiliations
  • University of Chicago (US)
  • Universitätsklinikum Aachen (DE)
  • University of Leeds (GB)
Corresponding author
  • Grossman, Robert L.
  • Pearson, Alexander T.
Venue
Nature Communications
Published
2021-07-20
Volume
12
Issue
1
Pages
4423-4423
ISSN
2041-1723 (ISSN-L)
OpenAlex
W3186679119
Cited by
330
References
83
Citations by year
2026: 85 · 2025: 55 · 2024: 71 · 2023: 60 · 2022: 47 · 2021: 11
FWCI
16.72
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RWTH Publications (RWTH Aachen) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • RWTH Publications (RWTH Aachen)
Funders
  • U.S. Department of Health & Human Services | NIH | National Institute of Dental and Craniofacial Research (K08-DE026500)
  • U.S. Department of Health & Human Services | NIH | National Cancer Institute (U01-CA243075)
  • Cancer Research Foundation
  • Adenoid Cystic Carcinoma Research Foundation
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Gene expression and cancer classification
Keywords
TCGA, tumor subtypes, survival prediction, tumor stage prediction, algorithmic fairness, quadratic programming, cross-site validation, color normalization, image augmentation
MeSH terms
Data Accuracy, Deep Learning, DNA Mutational Analysis, Humans, Image Processing, Computer-Assisted, Mutation, Neoplasm Staging, Neoplasms, Specimen Handling, Biomarkers, Tumor, Risk Assessment, Gene Expression Profiling

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The impact of site-specific digital histology signatures on deep learning model accuracy and bias. https://app.cochat.ai/papers/10.1038/s41467-021-24698-1
The paper
Howard, F. M., Dolezal, J., Kochanny, S., Schulte, J., Chen, H., Heij, L., Huo, D., Nanda, R., Olopade, O. I., Kather, J. N., Cipriani, N., Grossman, R. L., & Pearson, A. T. (2021). The impact of site-specific digital histology signatures on deep learning model accuracy and bias. Nature Communications. https://doi.org/10.1038/s41467-021-24698-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.