CoChat
Sign Up
Cited by 21AI in cancer detectionRadiomics and Machine Learning in Medical ImagingCancer Immunotherapy and Biomarkers

The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.

Comparing deep learning and pathologist quantification of cell-level PD-L1 expression in non-small cell lung cancer whole-slide images

Open access
van Eekelen, Leander, Spronck, Joey, Looijen-Salamon, Monika, Vos, Shoko, Munari, Enrico, Girolami, Ilaria, Eccher, Albino, Acs, Balazs, Boyaci, Ceren, de Souza, Gabriel Silva, Demirel-Andishmand, Muradije, Meesters, Luca Dulce, Zegers, Daan, van der Woude, Lieke, Theelen, Willemijn, van den Heuvel, Michel, Grünberg, Katrien, van Ginneken, Bram, van der Laak, Jeroen, Ciompi, Francesco·2024·Scientific Reports
DOI 10.1038/s41598-024-57067-1PMC PMC10965947
Open this paper in CoChat — free accountOpen PDF

Abstract

Programmed death-ligand 1 (PD-L1) expression is currently used in the clinic to assess eligibility for immune-checkpoint inhibitors via the tumor proportion score (TPS), but its efficacy is limited by high interobserver variability. Multiple papers have presented systems for the automatic quantification of TPS, but none report on the task of determining cell-level PD-L1 expression and often reserve their evaluation to a single PD-L1 monoclonal antibody or clinical center. In this paper, we report on a deep learning algorithm for detecting PD-L1 negative and positive tumor cells at a cellular level and evaluate it on a cell-level reference standard established by six readers on a multi-centric, multi PD-L1 assay dataset. This reference standard also provides for the first time a benchmark for computer vision algorithms. In addition, in line with other papers, we also evaluate our algorithm at slide-level by measuring the agreement between the algorithm and six pathologists on TPS quantification. We find a moderately low interobserver agreement at cell-level level (mean reader-reader F1 score = 0.68) which our algorithm sits slightly under (mean reader-AI F1 score = 0.55), especially for cases from the clinical center not included in the training set. Despite this, we find good AI-pathologist agreement on quantifying TPS compared to the interobserver agreement (mean reader-reader Cohen’s kappa = 0.54, 95% CI 0.26–0.81, mean reader-AI kappa = 0.49, 95% CI 0.27—0.72). In conclusion, our deep learning algorithm demonstrates promise in detecting PD-L1 expression at a cellular level and exhibits favorable agreement with pathologists in quantifying the tumor proportion score (TPS). We publicly release our models for use via the Grand-Challenge platform.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-024-57067-1

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Overview of study design. In (A), the model development process is shown. First, a nuclei detector is trained using point annotated nuclei. Secondly, polygons are drawn in regions of homogeneous cell types and intersected with detected nuclei. The nuclei are given the label of the corresponding region, resulting in multi-class labeled nuclei usable for training a multi-class PD-L1 detector. The PD-L1 detector is validated by (B) correlating slide-level tumor proportion score as determined by the system with the visual estimate of pathologists and (C) comparing its detection output to manual cell-level annotations of pathologists.
Figure 1 Overview of study design. In (A), the model development process is shown. First, a nuclei detector is trained using point annotated nuclei. Secondly, polygons are drawn in regions of homogeneous cell types and intersected with detected nuclei. The nuclei are given the label of the corresponding region, resulting in multi-class labeled nuclei usable for training a multi-class PD-L1 detector. The PD-L1 detector is validated by (B) correlating slide-level tumor proportion score as determined by the system with the visual estimate of pathologists and (C) comparing its detection output to manual cell-level annotations of pathologists.CC BY · PubMed Central
Figure 2 Validation of the nuclei detector. In (A), pairwise F1 scores are shown over cell predictions in the RUMC nuclei detection test set made by the nuclei detector and human readers. In (B), one ROI from the test set is shown with annotations by the readers and predictions from the nuclei detector overlaid on the PD-L1 immunohistochemistry. Annotations from the readers are shown as pink, light yellow, green and blue dots, while the AI predictions are shown as orange triangles.
Figure 2 Validation of the nuclei detector. In (A), pairwise F1 scores are shown over cell predictions in the RUMC nuclei detection test set made by the nuclei detector and human readers. In (B), one ROI from the test set is shown with annotations by the readers and predictions from the nuclei detector overlaid on the PD-L1 immunohistochemistry. Annotations from the readers are shown as pink, light yellow, green and blue dots, while the AI predictions are shown as orange triangles.CC BY · PubMed Central
Figure 3 An overview of PD-L1 detector results as compared to the cell-level predictions of the readers. Each reader in the middle column has their own unique symbol (diamond, square or circle), while the PD-L1 detector results are always shown as triangles in the last column. The color of the symbol indicates the predicted class: blue for PD-L1 negative tumor cells, red for PD-L1 positive tumor cells, yellow for other cells. The mean F1 scores for all reader-reader (‘inter-reader’) and reader-AI pairs are shown per ROI underneath each row.
Figure 3 An overview of PD-L1 detector results as compared to the cell-level predictions of the readers. Each reader in the middle column has their own unique symbol (diamond, square or circle), while the PD-L1 detector results are always shown as triangles in the last column. The color of the symbol indicates the predicted class: blue for PD-L1 negative tumor cells, red for PD-L1 positive tumor cells, yellow for other cells. The mean F1 scores for all reader-reader (‘inter-reader’) and reader-AI pairs are shown per ROI underneath each row.CC BY · PubMed Central
Figure 4 Pairwise F1 scores over the cases from the cell-level reader study, measuring the cell-level interobserver agreement of pathologists and AI on determining PD-L1 expression. The F1 scores are shown per clinical (RUMC, NEG and NKI) and are averaged over the three classes (PD-L1 negative/positive tumor cells and ‘other cells’). The F1 scores per class are reported in Supplementary Fig. 2.
Figure 4 Pairwise F1 scores over the cases from the cell-level reader study, measuring the cell-level interobserver agreement of pathologists and AI on determining PD-L1 expression. The F1 scores are shown per clinical (RUMC, NEG and NKI) and are averaged over the three classes (PD-L1 negative/positive tumor cells and ‘other cells’). The F1 scores per class are reported in Supplementary Fig. 2.CC BY · PubMed Central
Figure 5 The distribution of the linear Cohen’s kappa agreement of readers with other readers for predicting the tumor proportion score (TPS) on n = 100 test set cases. For this analysis, agreement on TPS was considered solely at the 1% and at the 50% cutoff. The diamonds in each boxplot visualize the agreement of the reader with the majority vote of human readers for that particular cutoff. The boxplots of each reader are sorted on the x-axis according to the median agreement with the majority vote over both cutoffs.
Figure 5 The distribution of the linear Cohen’s kappa agreement of readers with other readers for predicting the tumor proportion score (TPS) on n = 100 test set cases. For this analysis, agreement on TPS was considered solely at the 1% and at the 50% cutoff. The diamonds in each boxplot visualize the agreement of the reader with the majority vote of human readers for that particular cutoff. The boxplots of each reader are sorted on the x-axis according to the median agreement with the majority vote over both cutoffs.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 23

  1. Onoi, K. et al. Immune checkpoint inhibitors for lung cancer treatment: A review. J. Clin. Med. 9, 1362 (2020). · DOI
  2. Nasser, N. J., Gorenberg, M. & Agbarya, A. First line immunotherapy for non-small cell lung cancer. Pharmaceuticals 13, 373 (2020). · DOI
  3. Prelaj, A. et al. Predictive biomarkers of response for immune checkpoint inhibitors in non–small-cell lung cancer. Eur. J. Cancer 106, 144–159 (2019). · DOI
  4. Paolino, G. et al. PD-L1 evaluation in head and neck squamous cell carcinoma: Insights regarding specimens, heterogeneity and therapy. Pathol. Res. Pract. 226, 153605 (2021). · DOI
  5. Uruga, H. & Mino-Kenudson, M. Predictive biomarkers for response to immune checkpoint inhibitors in lung cancer: PD-L1 and beyond. Virchows Arch. 478, 31–44 (2021). · DOI
  6. Girolami, I. et al. Challenges facing pathologists evaluating PD-L1 in head & neck squamous cell carcinoma. J. Oral Pathol. 50, 864–873 (2021). · DOI
  7. Butter, R. et al. The impact of a pathologist’s personality on the interobserver variability and diagnostic accuracy of predictive PD-L1 immunohistochemistry in lung cancer. Lung Cancer 166, 143–149 (2022). · DOI
  8. Cooper, W. A. et al. Intra- and interobserver reproducibility assessment of PD-L1 biomarker in non–small cell lung cancer. Clin. Cancer Res. 23, 4569–4577 (2017). · DOI
  9. Lu, S. et al. Comparison of biomarker modalities for predicting response to PD-1/PD-L1 checkpoint blockade: A systematic review and meta-analysis. JAMA Oncol. 5, 1195–1204 (2019). · DOI
  10. Graham, S. et al. Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med. Image Anal. 58, 101563 (2019). · DOI
  11. Verma, R. et al. MoNuSAC2020: A multi-organ nuclei segmentation and classification challenge. IEEE Trans. Med. Imaging 40, 3413–3423 (2021). · DOI
  12. Hondelink, L. M. et al. Development and validation of a supervised deep learning algorithm for automated whole-slide programmed death-ligand 1 tumour proportion score assessment in non-small cell lung cancer. Histopathology 80, 635–647 (2022). · DOI
  13. Liu, J. et al. Automated tumor proportion score analysis for PD-L1 (22C3) expression in lung squamous cell carcinoma. Sci. Rep. 11, 1–9 (2021). · DOI
  14. Taylor, C. R. et al. A multi-institutional study to evaluate automated whole slide scoring of immunohistochemistry for assessment of programmed death-ligand 1 (PD-L1) expression in non-small cell lung cancer. Appl. Immunohistochem. Mol. Morphol. 27, 263 (2019). · DOI
  15. Choi, S. et al. Artificial intelligence–powered programmed death ligand 1 analyser reduces interobserver variation in tumour proportion score for non–small cell lung cancer with better prediction of immunotherapy response. Eur. J. Cancer 170, 17–26 (2022). · DOI
  16. Widmaier, M. et al. Comparison of continuous measures across diagnostic PD-L1 assays in non-small cell lung cancer using automated image analysis. Mod. Pathol. 33, 380–390 (2020). · DOI
  17. Wu, J. et al. Artificial intelligence-assisted system for precision diagnosis of PD-L1 expression in non-small cell lung cancer. Mod. Pathol. 35, 403–411 (2021). · DOI
  18. Munari, E. et al. PD-L1 assays 22C3 and SP263 are not interchangeable in non-small cell lung cancer when considering clinically relevant cutoffs. Am. J. Surg. Pathol. 42, 1384–1389 (2018). · DOI
  19. Munari, E. et al. Impact of PD-L1 and PD-1 expression on the prognostic significance of CD8+ tumor-infiltrating lymphocytes in non-small cell lung cancer. Front. Immunol. 1, 1971 (2021).
  20. Theelen, W. S. M. E. et al. Effect of pembrolizumab after stereotactic body radiotherapy vs pembrolizumab alone on tumor response in patients with advanced non-small cell lung cancer: Results of the PEMBRO-RT phase 2 randomized clinical trial. JAMA Oncol. 5, 1276–1282 (2019). · DOI
  21. Bándi, P., Balkenhol, M., Van Ginneken, B., Van Der Laak, J. & Litjens, G. Resolution-agnostic tissue segmentation in whole-slide histopathology images with convolutional neural networks. PeerJ 2019, 7 (2019).
  22. De-Bel, T. et al. Residual cyclegan for robust domain transformation of histopathological tissue slides. Med. Image Anal. 70, 102004 (2021). · DOI
  23. Kapil, A. et al. Domain adaptation-based deep learning for automated tumor cell (TC) scoring and survival analysis on PD-L1 stained tissue images. IEEE Trans. Med. Imaging 40, 2513–2523 (2021). · DOI

Cited by

Sorted by citations

See all 21

Metadata

Authors
van Eekelen, L., Spronck, J., Looijen-Salamon, M., Vos, S., Munari, E., Girolami, I., Eccher, A., Acs, B., Boyaci, C., de Souza, G. S., Demirel-Andishmand, M., Meesters, L. D., Zegers, D., van der Woude, L., Theelen, W., van den Heuvel, M., Grünberg, K., van Ginneken, B., van der Laak, J., & Ciompi, F.
Affiliations
  • Radboud University Medical Center (NL)
  • University of Brescia (IT)
  • Ospedale di Bolzano (IT)
  • University of Verona (IT)
  • The Netherlands Cancer Institute (NL)
Corresponding author
van Eekelen, Leander
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2024-03-26
Volume
14
Issue
1
Pages
7136-7136
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/s41598-024-57067-1
PMC
PMC10965947
OpenAlex
W4393192862
Cited by
21
References
2
Citations by year
2026: 5 · 2025: 11 · 2024: 5
FWCI
3.10
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Radboud Repository (Radboud University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Nederlandse Organisatie voor Wetenschappelijk Onderzoek (18388)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Health Sciences › Medicine › Oncology › Cancer Immunotherapy and Biomarkers
Keywords
PD-L1 expression, non-small cell lung cancer, tumor proportion score, whole slide images, deep learning, interobserver variability, immune checkpoint inhibitors
MeSH terms
Pathologists, Deep Learning, Carcinoma, Non-Small-Cell Lung, Humans, Immunohistochemistry, Lung Neoplasms, Biomarkers, Tumor, B7-H1 Antigen

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review