Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.

A whole-slide foundation model for digital pathology from real-world data

Open access

Abstract

Digital pathology poses unique computational challenges, as a standard gigapixel slide may comprise tens of thousands of image tiles 1–3 . Prior models have often resorted to subsampling a small portion of tiles for each slide, thus missing the important slide-level context 4 . Here we present Prov-GigaPath, a whole-slide pathology foundation model pretrained on 1.3 billion 256 × 256 pathology image tiles in 171,189 whole slides from Providence, a large US health network comprising 28 cancer centres. The slides originated from more than 30,000 patients covering 31 major tissue types. To pretrain Prov-GigaPath, we propose GigaPath, a novel vision transformer architecture for pretraining gigapixel pathology slides. To scale GigaPath for slide-level learning with tens of thousands of image tiles, GigaPath adapts the newly developed LongNet 5 method to digital pathology. To evaluate Prov-GigaPath, we construct a digital pathology benchmark comprising 9 cancer subtyping tasks and 17 pathomics tasks, using both Providence and TCGA data 6 . With large-scale pretraining and ultra-large-context modelling, Prov-GigaPath attains state-of-the-art performance on 25 out of 26 tasks, with significant improvement over the second-best method on 18 tasks. We further demonstrate the potential of Prov-GigaPath on vision–language pretraining for pathology 7,8 by incorporating the pathology reports. In sum, Prov-GigaPath is an open-weight foundation model that achieves state-of-the-art performance on various digital pathology tasks, demonstrating the importance of real-world data and whole-slide modelling.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41586-024-07441-w

Cites

60 of 60 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Overview of Prov-GigaPath. a, Flow chart showing the model architecture of Prov-GigaPath. Prov-GigaPath first serializes each input WSI into a sequence of 256 × 256 image tiles in row-major order and uses an image tile-level encoder to convert each image tile into a visual embedding. Then Prov-GigaPath applies a slide-level encoder based on the LongNet architecture to generate contextualized embeddings, which can serve as the basis for various downstream applications. b, Image tile-level pretraining using DINOv2. c, Slide-level pretraining with LongNet using masked autoencoder. [CLS] is the classification token.
Fig. 1 Overview of Prov-GigaPath. a, Flow chart showing the model architecture of Prov-GigaPath. Prov-GigaPath first serializes each input WSI into a sequence of 256 × 256 image tiles in row-major order and uses an image tile-level encoder to convert each image tile into a visual embedding. Then Prov-GigaPath applies a slide-level encoder based on the LongNet architecture to generate contextualized embeddings, which can serve as the basis for various downstream applications. b, Image tile-level pretraining using DINOv2. c, Slide-level pretraining with LongNet using masked autoencoder. [CLS] is the classification token.CC BY · PubMed Central
Fig. 2 Gene mutation prediction. a−j, Bar plots comparing the AUROC and AUPRC scores of Prov-GigaPath and competing methods on pan-cancer 18-biomarker (a,f), LUAD-specific 5-gene mutation prediction (b,g), pan-cancer 5-gene mutation prediction (c,h), LUAD-specific 5-gene mutation prediction on TCGA (d,i) and pan-cancer TMB prediction (e,j). k, Bar plot showing AUROC for each gene on LUAD-specific five-gene mutation prediction on TCGA. a−k, Data are mean ± s.e.m. across n = 10 independent experiments. The listed P value indicates the significance for Prov-GigaPath outperforming the best comparison approach, with one-sided Wilcoxon test. l, Comparison of AUROC scores for individual biomarkers in pan-cancer 18-biomarker predictions.
Fig. 2 Gene mutation prediction. a−j, Bar plots comparing the AUROC and AUPRC scores of Prov-GigaPath and competing methods on pan-cancer 18-biomarker (a,f), LUAD-specific 5-gene mutation prediction (b,g), pan-cancer 5-gene mutation prediction (c,h), LUAD-specific 5-gene mutation prediction on TCGA (d,i) and pan-cancer TMB prediction (e,j). k, Bar plot showing AUROC for each gene on LUAD-specific five-gene mutation prediction on TCGA. a−k, Data are mean ± s.e.m. across n = 10 independent experiments. The listed P value indicates the significance for Prov-GigaPath outperforming the best comparison approach, with one-sided Wilcoxon test. l, Comparison of AUROC scores for individual biomarkers in pan-cancer 18-biomarker predictions.CC BY · PubMed Central
Fig. 3 Comparison on cancer subtyping. a–f, Bar plots comparing cancer subtyping performance in terms of AUROC (a,c,e) and balanced accuracy (b,d,f) on nine cancer types. Data are mean ± s.e.m. across n = 10 independent experiments. The listed P value indicates the significance for Prov-GigaPath outperforming the best comparison approach, with one-sided Wilcoxon test. BACC, balanced accuracy. BRCA, breast invasive carcinoma; CNS, central nervous system; COADREAD, colorectal adenocarcinoma; DIFG, diffuse intrinsic pontine glioma; EGC, early gastric cancer; HB, hepatobiliary; NSCLC, non-small cell lung cancer; OVT, ovarian cancer; RCC, renal cell cancer.
Fig. 3 Comparison on cancer subtyping. a–f, Bar plots comparing cancer subtyping performance in terms of AUROC (a,c,e) and balanced accuracy (b,d,f) on nine cancer types. Data are mean ± s.e.m. across n = 10 independent experiments. The listed P value indicates the significance for Prov-GigaPath outperforming the best comparison approach, with one-sided Wilcoxon test. BACC, balanced accuracy. BRCA, breast invasive carcinoma; CNS, central nervous system; COADREAD, colorectal adenocarcinoma; DIFG, diffuse intrinsic pontine glioma; EGC, early gastric cancer; HB, hepatobiliary; NSCLC, non-small cell lung cancer; OVT, ovarian cancer; RCC, renal cell cancer.CC BY · PubMed Central
Fig. 4 Comparison on image–report alignment. a, Flow chart showing the fine-tuning of Prov-GigaPath using pathology reports. Real-world pathology reports are processed using GPT-3.5 from OpenAI to remove information irrelevant to cancer diagnosis. We performed the CLIP-based contrastive learning to align Prov-GigaPath and PubMedBERT. b, The fine-tuned Prov-GigaPath can then be used to perform zero-shot cancer subtyping and mutation prediction. The input of Prov-GigaPath is a sequence of tiles segmented from a WSI, and the inputs of the text encoder PubMedBERT are manually designed prompts representing cancer types and mutations. Based on the output of Prov-GigaPath and PubMedBERT, we can calculate the probability of the input WSI being classified into specific cancer subtypes and mutations. c, Bar plots comparing zero-shot subtyping performance on NSCLC and COADREAD in terms of BACC, precision and f1. d, Bar plots comparing the performance on mutation prediction using the fine-tuned model for six genes. c,d, Data are mean ± s.e.m. across n = 50 experiments. The listed P value indicates the significance for Prov-GigaPath outperforming the best comparison approach, with one-sided Wilcoxon test. e, Scatter plots comparing the performance between Prov-GigaPath and MI-Zero in terms of BACC on zero-shot cancer subtyping. Each dot indicates one trial with a particular set of text query formulations.
Fig. 4 Comparison on image–report alignment. a, Flow chart showing the fine-tuning of Prov-GigaPath using pathology reports. Real-world pathology reports are processed using GPT-3.5 from OpenAI to remove information irrelevant to cancer diagnosis. We performed the CLIP-based contrastive learning to align Prov-GigaPath and PubMedBERT. b, The fine-tuned Prov-GigaPath can then be used to perform zero-shot cancer subtyping and mutation prediction. The input of Prov-GigaPath is a sequence of tiles segmented from a WSI, and the inputs of the text encoder PubMedBERT are manually designed prompts representing cancer types and mutations. Based on the output of Prov-GigaPath and PubMedBERT, we can calculate the probability of the input WSI being classified into specific cancer subtypes and mutations. c, Bar plots comparing zero-shot subtyping performance on NSCLC and COADREAD in terms of BACC, precision and f1. d, Bar plots comparing the performance on mutation prediction using the fine-tuned model for six genes. c,d, Data are mean ± s.e.m. across n = 50 experiments. The listed P value indicates the significance for Prov-GigaPath outperforming the best comparison approach, with one-sided Wilcoxon test. e, Scatter plots comparing the performance between Prov-GigaPath and MI-Zero in terms of BACC on zero-shot cancer subtyping. Each dot indicates one trial with a particular set of text query formulations.CC BY · PubMed Central
Extended Data Fig. 1 Comparison on Pan-cancer 18-biomarker prediction. Bar plot showing the AUPRC score for each biomarker on the 18-biomarker prediction by Prov-GigaPath and competing methods.
Extended Data Fig. 1 Comparison on Pan-cancer 18-biomarker prediction. Bar plot showing the AUPRC score for each biomarker on the 18-biomarker prediction by Prov-GigaPath and competing methods.CC BY · PubMed Central
Extended Data Fig. 2 Comparison on LUAD 5-gene mutation prediction. Bar plots showing AUROC and AUPRC scores for predicting each gene mutation on LUAD 5-gene mutation prediction. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test.
Extended Data Fig. 2 Comparison on LUAD 5-gene mutation prediction. Bar plots showing AUROC and AUPRC scores for predicting each gene mutation on LUAD 5-gene mutation prediction. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test.CC BY · PubMed Central
Extended Data Fig. 3 Comparison on Pan-cancer 5-gene mutation prediction. Bar plots showing AUROC and AUPRC scores for predicting each gene mutation on Pan-cancer 5-gene mutation prediction. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test.
Extended Data Fig. 3 Comparison on Pan-cancer 5-gene mutation prediction. Bar plots showing AUROC and AUPRC scores for predicting each gene mutation on Pan-cancer 5-gene mutation prediction. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test.CC BY · PubMed Central
Extended Data Fig. 4 Comparison on LUAD 5-gene mutation prediction in TCGA. Bar plots showing AUPRC scores for predicting each gene mutation on LUAD 5-gene mutation prediction in TCGA. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test.
Extended Data Fig. 4 Comparison on LUAD 5-gene mutation prediction in TCGA. Bar plots showing AUPRC scores for predicting each gene mutation on LUAD 5-gene mutation prediction in TCGA. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test.CC BY · PubMed Central
Extended Data Fig. 5 Comparison on mutation prediction on new colorectal patients. Bar plots showing AUROC and AUPRC scores for predicting 5-gene mutation and TMB status on new patients from Providence. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test.
Extended Data Fig. 5 Comparison on mutation prediction on new colorectal patients. Bar plots showing AUROC and AUPRC scores for predicting 5-gene mutation and TMB status on new patients from Providence. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test.CC BY · PubMed Central
Extended Data Fig. 6 Comparison between pretraining the same model using Prov-Path and TCGA. a-b, Bar plots showing the AUROC (a) and AURPC (b) on LUAD 5-gene mutation prediction in TCGA using models trained on Prov-Path and TCGA. Prov-GigaPath is GigaPath trained on Prov-Path. GigaPath-TCGA is GigaPath trained on TCGA. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms GigaPath-TCGA, with one-sided Wilcoxon test.
Extended Data Fig. 6 Comparison between pretraining the same model using Prov-Path and TCGA. a-b, Bar plots showing the AUROC (a) and AURPC (b) on LUAD 5-gene mutation prediction in TCGA using models trained on Prov-Path and TCGA. Prov-GigaPath is GigaPath trained on Prov-Path. GigaPath-TCGA is GigaPath trained on TCGA. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms GigaPath-TCGA, with one-sided Wilcoxon test.CC BY · PubMed Central
Extended Data Fig. 7 Comparison between GigaPath trained using Prov-Path and HIPT trained using Prov-Path on mutation prediction. a-j: Bar plots showing the AUROC (a-e) and AURPC (f-j) of mutation prediction tasks by Prov-GigaPath and HIPT-Prov-Path. HIPT-Prov-Path indicates HIPT pretrained on Prov-Path. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the HIPT-Prov-Path, with one-sided Wilcoxon test.
Extended Data Fig. 7 Comparison between GigaPath trained using Prov-Path and HIPT trained using Prov-Path on mutation prediction. a-j: Bar plots showing the AUROC (a-e) and AURPC (f-j) of mutation prediction tasks by Prov-GigaPath and HIPT-Prov-Path. HIPT-Prov-Path indicates HIPT pretrained on Prov-Path. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the HIPT-Prov-Path, with one-sided Wilcoxon test.CC BY · PubMed Central
Extended Data Fig. 8 Comparison between GigaPath trained using Prov-Path and HIPT trained using Prov-Path on cancer subtyping. a-f, Bar plots showing the AUROC (a,c,e) and BACC (b,d,f) of cancer subtyping tasks by Prov-GigaPath and HIPT-Prov-Path. HIPT-Prov-Path indicates HIPT pretrained on Prov-Path. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the HIPT-Prov-Path, with one-sided Wilcoxon test.
Extended Data Fig. 8 Comparison between GigaPath trained using Prov-Path and HIPT trained using Prov-Path on cancer subtyping. a-f, Bar plots showing the AUROC (a,c,e) and BACC (b,d,f) of cancer subtyping tasks by Prov-GigaPath and HIPT-Prov-Path. HIPT-Prov-Path indicates HIPT pretrained on Prov-Path. The error bars show the standard error across n = 10 independent experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the HIPT-Prov-Path, with one-sided Wilcoxon test.CC BY · PubMed Central
Extended Data Fig. 9 Alignment between pathology reports and images. a-d, Bar plots showing the performance of f1 (a), Precision (b), AUROC (c) and AUPRC (d) using fine-tuned Prov-GigaPath to predict mutations in the zero-shot learning setting. The error bars show the standard error across n = 50 experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test. e, Scatter plots comparing Prov-GigaPath and MI-Zero on cancer subtyping prediction and mutation prediction in terms of balanced accuracy (BACC).
Extended Data Fig. 9 Alignment between pathology reports and images. a-d, Bar plots showing the performance of f1 (a), Precision (b), AUROC (c) and AUPRC (d) using fine-tuned Prov-GigaPath to predict mutations in the zero-shot learning setting. The error bars show the standard error across n = 50 experiments and the bar centre shows the mean value. The listed p-value indicates the significance level that Prov-GigaPath outperforms the best comparison approach, with one-sided Wilcoxon test. e, Scatter plots comparing Prov-GigaPath and MI-Zero on cancer subtyping prediction and mutation prediction in terms of balanced accuracy (BACC).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

5 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 55 references

Research works printed but not linked: 4

  1. Ilse, M., Tomczak, J. & Welling, M. Attention-based deep multiple instance learning. In Proc. 35th International Conference on Machine Learning (eds Dy, J. & Krause, A.) 2127–2136 (IMLS, 2018).
  2. Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (eds Daumé, H. & Singh, A.) 1597–1607 (PMLR, 2020).
  3. Kenton, J. D. M.-W. C. & Toutanova, L. K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT 2019 (eds Burstein, J. et al.) 4171–4186 (Association for Computational Linguistics, 2019).
  4. Bao, H., Dong, L., Piao, S. & Wei, F. BEIT: BERT pre-training of image transformers. In International Conference on Learning Representations (2021).

Cited by

See all 923

Metadata

Authors
Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y., Xu, Y., Wei, M., Wang, W., Ma, S., Wei, F., Yang, J., Li, C., Gao, J., Rosemon, J., Bower, T., Lee, S., Weerasinghe, R., Wright, B. J., Robicsek, A., Piening, B., Bifulco, C., Wang, S., & Poon, H.
Affiliations
  • Microsoft (United States) (US)
  • Providence Portland Medical Center (US)
  • University of Washington (US)
Corresponding author
  • Bifulco, Carlo
  • Wang, Sheng
  • Poon, Hoifung
Venue
Nature
Published
2024-05-22
Volume
630
Issue
8015
Pages
181-188
ISSN
  • 0028-0836 (ISSN-L)
  • 1476-4687
OpenAlex
W4398201291
Cited by
921
References
55
Citations by year
2026: 480 · 2025: 355 · 2024: 73
FWCI
134.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
digital pathology, whole slide images, foundation models, vision transformer, cancer subtyping, pathomics, vision-language pre-training, pathology reports, open-weight models, real-world data
MeSH terms
Machine Learning, Female, Humans, Image Processing, Computer-Assisted, Male, Neoplasms, Pathology, Clinical, Benchmarking, Datasets as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A whole-slide foundation model for digital pathology from real-world data. https://app.cochat.ai/papers/10.1038/s41586-024-07441-w
The paper
Xu, H., Usuyama, N., Bagga, J., Zhang, S., Rao, R., Naumann, T., Wong, C., Gero, Z., González, J., Gu, Y., Xu, Y., Wei, M., Wang, W., Ma, S., Wei, F., Yang, J., Li, C., Gao, J., Rosemon, J., . . . Poon, H. (2024). A whole-slide foundation model for digital pathology from real-world data. Nature. https://doi.org/10.1038/s41586-024-07441-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.