Deep learning-based image analysis predicts PD-L1 status from H&E-stained histopathology images in breast cancer
Open accessAbstract
Programmed death ligand-1 (PD-L1) has been recently adopted for breast cancer as a predictive biomarker for immunotherapies. The cost, time, and variability of PD-L1 quantification by immunohistochemistry (IHC) are a challenge. In contrast, hematoxylin and eosin (H&E) is a robust staining used routinely for cancer diagnosis. Here, we show that PD-L1 expression can be predicted from H&E-stained images by employing state-of-the-art deep learning techniques. With the help of two expert pathologists and a designed annotation software, we construct a dataset to assess the feasibility of PD-L1 prediction from H&E in breast cancer. In a cohort of 3,376 patients, our system predicts the PD-L1 status in a high area under the curve (AUC) of 0.91 - 0.93. Our system is validated on two external datasets, including an independent clinical trial cohort, showing consistent prediction performance. Furthermore, the proposed system predicts which cases are prone to pathologists miss-interpretation, showing it can serve as a decision support and quality assurance system in clinical practice.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Other efforts have used AI on H&E images to predict TIL presence and PD-L1 expression status in breast cancer.
“Other efforts have used AI on H&E images to directly predict TIL presence (85,86), a notable prognostic and predictive biomarker in breast cancer, and programmed cell death ligand 1 (PD-L1) expression status, a predictive biomarker for immunotherapies (87).”
From Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions (2024) · cites this paper as [87]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Bray, F., Ferlay, J., Soerjomataram, I., Siegel, R. L., Torre, L. A., & Jemal, A. (2018) · CA: A Cancer Journal for Clinicians · doi:10.3322/caac.21492 · Cited by 84,109 · see citing works
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858826 · Cited by 9,556 · see citing works
Weiss, K., Khoshgoftaar, T. M., & Wang, D. (2016) · Journal of Big Data · doi:10.1186/s40537-016-0043-6 · Cited by 6,260 · see citing works
Clinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799 · see citing works
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524 · see citing works
Quantification of histochemical staining by color deconvolution.
Ruifrok, A. C. C., & Johnston, D. A. (2001) · PubMed · Cited by 2,029 · see citing works
Schmid, P., Rugo, H. S., Adams, S., Schneeweiss, A., Barrios, C. H., Iwata, H., Diéras, V., Henschel, V., Molinero, L., Chui, S. Y., et al. (2020) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30689-8 · Cited by 1,216 · see citing works
Yu, K.-H., Zhang, C., Berry, G. J., Altman, R. B., Ré, C., Rubin, D. L., & Snyder, M. (2016) · Nature Communications · doi:10.1038/ncomms12474 · Cited by 972 · see citing works
Systematic Analysis of Breast Cancer Morphology Uncovers Stromal Features Associated with Survival
Beck, A. H., Sangoi, A. R., Leung, S. C. Y., Marinelli, R. J., Nielsen, T. O., van de Vijver, M. J., West, R. B., van de Rijn, M., & Koller, D. (2011) · Science Translational Medicine · doi:10.1126/scitranslmed.3002564 · Cited by 765 · see citing works
Rimm, D. L., Han, G., Taube, J. M., Yi, E. S., Bridge, J. A., Flieder, D. B., Homer, R., West, W. W., Wu, H., Roden, A. C., et al. (2017) · JAMA Oncology · doi:10.1001/jamaoncol.2017.0013 · Cited by 764 · see citing works
PDL1 Regulation by p53 via miR-34
Cortez, M. A., Ivan, C., Valdecanas, D., Wang, X., Peltier, H. J., Ye, Y., Araujo, L., Carbone, D. P., Shilo, K., Giri, D. K., et al. (2016) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/djv303 · Cited by 683 · see citing works
Adams, S., Loi, S., Toppmeyer, D., Cescon, D. W., Laurentiis, M. D., Nanda, R., Winer, E. P., Mukai, H., Tamura, K., Armstrong, A. C., et al. (2018) · Annals of Oncology · doi:10.1093/annonc/mdy518 · Cited by 632 · see citing works
Prevalence and mutational determinants of high tumor mutation burden in breast cancer
Barroso‐Sousa, R., Jain, E., Cohen, O., Kim, D., Buendia-Buendia, J. E., Winer, E. P., Lin, N., Tolaney, S. M., & Wagle, N. (2020) · Annals of Oncology · doi:10.1016/j.annonc.2019.11.010 · Cited by 363 · see citing works
Pantanowitz, L., Quiroga‐Garza, G. M., Bien, L., Heled, R., Laifenfeld, D., Linhart, C., Sandbank, J., Shach, A. A., Shalev, V., Vecsler, M., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30159-x · Cited by 361 · see citing works
Neoantigen screening identifies broad TP53 mutant immunogenicity in patients with epithelial cancers
Malekzadeh, P., Pasetto, A., Robbins, P. F., Parkhurst, M. R., Paria, B. C., Jia, L., Gartner, J. J., Hill, V. K., Yu, Z., Restifo, N. P., et al. (2019) · Journal of Clinical Investigation · doi:10.1172/jci123791 · Cited by 274 · see citing works
Naik, N., Madani, A., Esteva, A., Keskar, N. S., Press, M. F., Ruderman, D., Agus, D. B., & Socher, R. (2020) · Nature Communications · doi:10.1038/s41467-020-19334-3 · Cited by 253 · see citing works
PD-1/PD-L1 Targeting in Breast Cancer: The First Clinical Evidences are Emerging—A Literature Review
Planes-Laine, G., Rochigneux, P., Bertucci, F., Chrétien, A.-S., Viens, P., Sabatier, R., & Gonçalvès, A. (2019) · Cancers · doi:10.3390/cancers11071033 · Cited by 220 · see citing works
Li, Z., Dong, P., Ren, M., Song, Y., Qian, X., Yang, Y., Li, S., Zhang, X., & Liu, F. (2016) · Journal of Cancer · doi:10.7150/jca.14549 · Cited by 158 · see citing works
Determining breast cancer biomarker status and associated morphological features using deep learning
Gamble, P., Jaroensri, R., Wang, H., Tan, F., Moran, M., Brown, T., Flament-Auvigne, I., Rakha, E. A., Toss, M., Dabbs, D. J., et al. (2021) · Communications Medicine · doi:10.1038/s43856-021-00013-3 · Cited by 154 · see citing works
Shamai, G., Binenbaum, Y., Slossberg, R., Duek, I., Gil, Z., & Kimmel, R. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.7700 · Cited by 133 · see citing works
Self-citationReisenbichler, E., Han, G., Bellizzi, A. M., Bossuyt, V., Brock, J. E., Cole, K. S., Fadare, O., Hameed, O., Hanley, K. Z., Harrison, B. T., et al. (2020) · Modern Pathology · doi:10.1038/s41379-020-0544-x · Cited by 132 · see citing works
Sha, L., Osinski, B. L., Ho, I. Y., Tan, T. L., Willis, C., Weiss, H., Beaubier, N., Mahon, B. M., Taxter, T. J., & Yip, S. S. F. (2019) · Journal of Pathology Informatics · doi:10.4103/jpi.jpi_24_19 · Cited by 130 · see citing works
Kapil, A., Meier, A., Zuraw, A., Steele, K. E., Rebelatto, M. C., Schmidt, G., & Brieu, N. (2018) · Scientific Reports · doi:10.1038/s41598-018-35501-5 · Cited by 110 · see citing works
Deep learned tissue “fingerprints” classify breast cancers by ER/PR/Her2 status from H&E images
Rawat, R. R., Ortega, I., Roy, P., Sha, F., Shibata, D., Ruderman, D., & Agus, D. B. (2020) · Scientific Reports · doi:10.1038/s41598-020-64156-4 · Cited by 100 · see citing works
Cyprian, F. S., Akhtar, S., Gatalica, Z., & Vranić, S. (2019) · Bosnian Journal of Basic Medical Sciences · doi:10.17305/bjbms.2019.4204 · Cited by 99 · see citing works
Akinboro, O., Larkins, E., Pai-Scherf, L. H., Mathieu, L. N., Ren, Y., Cheng, J., Fiero, M. H., Fu, W., Bi, Y., Kalavar, S., et al. (2022) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-21-3844 · Cited by 84 · see citing works
Bychkov, D., Linder, N., Tiulpin, A., Kücükel, H., Lundin, M., Nordling, S., Sihto, H. J., Isola, J. J., Lehtimäki, T., Kellokumpu‐Lehtinen, P., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-83102-6 · Cited by 76 · see citing works
Humphries, M. P., Hynes, S. O., Bingham, V., Cougot, D., James, J. A., Patell-Socha, F., Parkes, E. E., Blayney, J. K., O’Rorke, M. A., Irwin, G. W., et al. (2018) · Journal of Oncology · doi:10.1155/2018/2937012 · Cited by 62 · see citing works
Widmaier, M., Wiestler, T., Walker, J., et al. (2019) · Modern Pathology · doi:10.1038/s41379-019-0349-y · Cited by 59 · see citing works
Polónia, A., Pinto, R., Cameselle‐Teijeiro, J., Schmitt, F. C., & Paredes, J. (2017) · Journal of Clinical Pathology · doi:10.1136/jclinpath-2016-203990 · Cited by 54 · see citing works
Self-citationCho, Y. A., Lee, H., Kim, D. G., Kim, H., Ha, S. Y., Choi, Y., Jang, K., & Kim, K. (2021) · Cancers · doi:10.3390/cancers13184659 · Cited by 51 · see citing works
Automated PD-L1 Scoring Using Artificial Intelligence in Head and Neck Squamous Cell Carcinoma
Puladi, B., Ooms, M., Kintsler, S., Houschyar, K. S., Steib, F., Modabber, A., Hölzle, F., Knüchel, R., & Braunschweig, T. A. (2021) · Cancers · doi:10.3390/cancers13174409 · Cited by 47 · see citing works
Wang, X., Wang, L., Bu, H., Zhang, N., Yue, M., Jia, Z., Cai, L., He, J., Wang, Y., Xu, X., et al. (2021) · npj Breast Cancer · doi:10.1038/s41523-021-00268-y · Cited by 40 · see citing works
Pang, J., Castles, B., Byrne, D. J., et al. (2021) · The American Journal of Surgical Pathology · doi:10.1097/pas.0000000000001701 · Cited by 39 · see citing works
Nguyen, H.-G., Blank, A., Dawson, H., Lugli, A., & Zlobec, I. (2021) · Scientific Reports · doi:10.1038/s41598-021-81352-y · Cited by 29 · see citing works
Generating Region of Interests for Invasive Breast Cancer in Histopathological Whole-Slide-Image
Patil, S. M., Li, T., & Wang, M. D. (2020) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac48688.2020.0-174 · Cited by 28 · see citing works
Hoda, R. S., Brogi, E., D’Alfonso, T. M., Grabenstetter, A., Giri, D. D., Hanna, M. G., Kuba, M. G., Murray, M. P., Vallejo, C. E., Zhang, H., et al. (2021) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2020-0451-oa · Cited by 22 · see citing works
Batenchuk, C., Chang, H.-W., Cimermančič, P., et al. (2020) · Journal of Clinical Oncology · doi:10.1200/jco.2020.38.15_suppl.3122 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [3]Humphries MP A J. Oncol. 2018 2018 2937012 30651729 PMC6311859
- [37]Gulum MA Trombley CM Kantardzic M A review of explainable deep learning cancer detection models in medical imaging NATO Adv. Sci. Inst. Ser. E Appl. Sci. 2021 11 4573
- [39]Selvaraju, R. R., Cogswell, M. & Das, A. Grad-cam: Visual explanations from deep networks via gradient-based localization. Proc. Estonian Acad. Sci. Biol. Ecol. https://arxiv.org/abs/1610.02391. · arXiv
- [41]Roche, V. Ventana PD-L1 (SP142) Assay interpretation guide for triple-negative breast carcinoma (TNBC). https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=Ventana+PD-L1+%28SP142%29+Assay+interpretation+guide+for+triple-negative+breast+carcinoma+%28TNBC%29&btnG= (2019).
- [44]Liu et al. On the variance of the adaptive learning rate and beyond. Preprint at arXiv:1908. 03265.
Cited by
Sorted by citations
Artificial Intelligence for Predictive Biomarker Discovery in Immuno-Oncology: A Systematic Review
Prelaj, A., Miskovic, V., Zanitti, M., Trovo, F., Genova, C., Viscardi, G., Rebuzzi, S. E., Mazzeo, L., Provenzano, L., Kosta, S., et al. (2024) · Annals of Oncology · doi:10.1016/j.annonc.2023.10.125 · Cited by 340 · see citing works
Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions
Lotter, W., Hassett, M. J., Schultz, N., Kehl, K. L., Van Allen, E. M., & Cerami, E. (2024) · Cancer Discovery · doi:10.1158/2159-8290.cd-23-1199 · Cited by 196 · see citing works
Machine learning methods for histopathological image analysis: updates in 2024
Komura, D., Ochi, M., & Ishikawa, S. (2025) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2024.12.033 · Cited by 76 · see citing works
Wang, F., Xu, H., Wang, X., Wu, M., Ding, S., Duan, L., Yang, C., Li, J., Ge, X., Qin, Y., et al. (2026) · iScience · doi:10.1016/j.isci.2026.115155
“5,6 More importantly, existing methods struggle to effectively integrate molecular-level cell type annotations with tissue-level spatial morphological features, leading to incomplete and mechanistically ambiguous characterization of the immune microenvironment.”
- Authors
- Shamai, G., Livne, A., Polónia, A., Sabo, E., Cretu, A., Bar-Sela, G., & Kimmel, R.
- Affiliations
- Technion – Israel Institute of Technology (IL)
- Universidade do Porto (PT)
- Carmel Medical Center (IL)
- Corresponding author
- Shamai, Gil
- Venue
- Nature Communications
- Published
- 2022-11-08
- Volume
- 13
- Issue
- 1
- Pages
- 6753-6753
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC9643479
- OpenAlex
- W4308573933
- Cited by
- 193
- References
- 40
- Citations by year
- 2026: 38 · 2025: 80 · 2024: 55 · 2023: 19 · 2022: 1
- FWCI
- 12.70
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Nature Communications (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Israel Science Foundation (3864/21, 679/18)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Health Sciences › Medicine › Oncology › Cancer Immunotherapy and Biomarkers
- Keywords
- breast cancer, deep learning, immunotherapy predictive biomarker, immunohistochemistry alternative, annotation tools, decision support system, quality assurance in pathology
- MeSH terms
- Deep Learning, Breast Neoplasms, Female, Hematoxylin, Humans, Lung Neoplasms, Staining and Labeling, Biomarkers, Tumor, B7-H1 Antigen
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
