CoChat
Sign Up
Cited by 82AI in cancer detectionRadiomics and Machine Learning in Medical ImagingBrain Tumor Detection and Classification

Vision transformer-based weakly supervised histopathological image analysis of primary brain tumors

Open access
Li, Zhongxiao, Cong, Yuwei, Chen, Xin, Qi, Jiping, Sun, Jingxian, Yan, Tao, Yang, He, Liu, Junsi +15 more·2023·iScience
DOI 10.1016/j.isci.2022.105872PMC PMC9839963
Open this paper in CoChat — free account

Abstract

Diagnosis of primary brain tumors relies heavily on histopathology. Although various computational pathology methods have been developed for automated diagnosis of primary brain tumors, they usually require neuropathologists' annotation of region of interests or selection of image patches on whole-slide images (WSI). We developed an end-to-end Vision Transformer (ViT) – based deep learning architecture for brain tumor WSI analysis, yielding a highly interpretable deep-learning model, ViT-WSI. Based on the principle of weakly supervised machine learning, ViT-WSI accomplishes the task of major primary brain tumor type and subtype classification. Using a systematic gradient-based attribution analysis procedure, ViT-WSI can discover diagnostic histopathological features for primary brain tumors. Furthermore, we demonstrated that ViT-WSI has high predictive power of inferring the status of three diagnostic glioma molecular markers, IDH1 mutation, p53 mutation, and MGMT methylation, directly from H&E-stained histopathological images, with patient level AUC scores of 0.960, 0.874, and 0.845, respectively.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.isci.2022.105872

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The development of the vision transformer (ViT) enabled a self-attention manner to aggregate the patch embeddings and position connectivity in the WSI, as well as the hybrid model Graph-Transformer.

    “Despite the traditional CNN method [95], the development of the vision transformer (ViT) [96] enabled a self-attention manner to aggregate the patch embeddings and position connectivity in the WSI [97], as well as the hybrid model “Graph-Transformer”.”

    From Challenges in AI-driven Biomedical Multimodal Data Fusion and Analysis (2025) · cites this paper as [97]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 60

  1. [1]Van den Bent M.J. Interobserver variation of the histopathological diagnosis in clinical trials on glioma: a clinician’s perspective Acta Neuropathol. 120 2010 297 304 20644945 PMC2910894
  2. [2]Willis J. Smith C. Ironside J.W. Erridge S. Whittle I.R. Everington D. The accuracy of meningioma grading: a 10-year retrospective audit Neuropathol. Appl. Neurobiol. 31 2005 141 149 15771707
  3. [3]Louis D.N. Perry A. Reifenberger G. von Deimling A. Figarella-Branger D. Cavenee W.K. Ohgaki H. Wiestler O.D. Kleihues P. Ellison D.W. The 2016 world health organization classification of tumors of the central nervous system: a summary Acta Neuropathol. 131 2016 803 820 10.1007/s00401-016-1545-1 27157931 · DOI
  4. [4]Rapôso C. Vitorino-Araujo J.L. Barreto N. Gliomas W.D. Molecular Markers of Gliomas to Predict Treatment and Prognosis: Current State and Future Directions 2021 Exon Publications 10.36255/exonpublications.gliomas.2021 chapter10 34038057 · DOI
  5. [5]Fraggetta F. Garozzo S. Zannoni G.F. Pantanowitz L. Rossi E.D. Routine digital pathology workflow: the Catania experience J. Pathol. Inf. 8 2017 51 PMC5760840 29416914
  6. [6]Litjens G. Bandi P. Ehteshami Bejnordi B. Geessink O. Balkenhol M. Bult P. Halilovic A. Hermsen M. van de Loo R. Vogels R. 1399 H& E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset GigaScience 7 2018 giy065 10.1093/gigascience/giy065 29860392 PMC6007545 · DOI
  7. [7]Couture H.D. Williams L.A. Geradts J. Nyante S.J. Butler E.N. Marron J.S. Perou C.M. Troester M.A. Niethammer M. Image analysis with deep learning to predict breast cancer grade, ER status, histologic subtype, and intrinsic subtype npj Breast Cancer 4 2018 30 10.1038/s41523-018-0079-1 30182055 PMC6120869 · DOI · Paper page
  8. [8]Strom P. Kartasalo K. Olsson H. Artificial intelligence for diagnosis and grading of prostate cancer in biopsies: a population-based, diagnostic study Lancet Oncol. 21 2020 E70 31926806
  9. [9]Courtiol P. Maussion C. Moarii M. Pronier E. Pilcer S. Sefta M. Manceron P. Toldo S. Zaslavskiy M. Le Stang N. Deep learning-based classification of mesothelioma improves prediction of patient outcome Nat. Med. 25 2019 1519 1525 31591589
  10. [10]Coudray N. Ocampo P.S. Sakellaropoulos T. Narula N. Snuderl M. Fenyö D. Moreira A.L. Razavian N. Tsirigos A. Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning Nat. Med. 24 2018 1559 1567 10.1038/s41591-018-0177-5 30224757 PMC9847512 · DOI · Paper page
  11. [11]Huang K.K. Huang J. Wu J.K.L. Lee M. Tay S.T. Kumar V. Ramnarayanan K. Padmanabhan N. Xu C. Tan A.L.K. Long-read transcriptome sequencing reveals abundant promoter diversity in distinct molecular subtypes of gastric cancer Genome Biol. 22 2021 44 33482911 PMC7821541
  12. [12]Jain M.S. Massoud T.F. Predicting tumour mutational burden from histopathological images using multiscale deep learning Nat. Mach. Intell. 2 2020 356 362 10.1038/s42256-020-0190-5 · DOI · Paper page
  13. [14]Campanella G. Hanna M.G. Geneslaw L. Miraflor A. Werneck Krauss Silva V. Busam K.J. Brogi E. Reuter V.E. Klimstra D.S. Fuchs T.J. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images Nat. Med. 25 2019 1301 1309 10.1038/s41591-019-0508-1 31308507 PMC7418463 · DOI · Paper page
  14. [15]Lu M.Y. Williamson D.F.K. Chen T.Y. Chen R.J. Barbieri M. Mahmood F. Data-efficient and weakly supervised computational pathology on whole-slide images Nat. Biomed. Eng. 5 2021 555 570 10.1038/s41551-020-00682-w 33649564 PMC8711640 · DOI · Paper page
  15. [16]Lu M.Y. Chen T.Y. Williamson D.F.K. Zhao M. Shady M. Lipkova J. Mahmood F. AI-based pathology predicts origins for cancers of unknown primary Nature 594 2021 106 110 10.1038/s41586-021-03512-4 33953404 · DOI · Paper page
  16. [17]Ostrom Q.T. Patil N. Cioffi G. Waite K. Kruchko C. Barnholtz-Sloan J.S. CBTRUS statistical report: primary brain and other central nervous system tumors diagnosed in the United States in 2013–2017 Neuro Oncol. 22 2020 iv1 iv96 10.1093/neuonc/noaa200 33123732 PMC7596247 · DOI
  17. [19]Ertosun M.G. Rubin D.L. Automated Grading of Gliomas Using Deep Learning in Digital Pathology Images: A Modular Approach with Ensemble of Convolutional Neural Networks 2015 American Medical Informatics Association 1899 PMC4765616 26958289
  18. [20]Jin L. Shi F. Chun Q. Chen H. Ma Y. Wu S. Hameed N.U.F. Mei C. Lu J. Zhang J. Artificial intelligence neuropathologist for glioma classification using deep learning on hematoxylin and eosin stained slide images and molecular markers Neuro Oncol. 23 2021 44 52 10.1093/neuonc/noaa163 32663285 PMC7850049 · DOI
  19. [21]Liu M. Zhang J. Lian C. Shen D. Weakly supervised deep learning for brain disease prognosis using MRI and incomplete clinical scores IEEE Trans. Cybern. 50 2020 3381 3392 30932861 PMC8034591
  20. [22]Ji Z. Shen Y. Ma C. Gao M. Scribble-based Hierarchical Weakly Supervised Learning for Brain Tumor Segmentation 2019 Springer 175 183
  21. [25]Ridnik T. Baruch E.B. Noy A. Zelnik-Manor L. ImageNet-21K pretraining for the masses Preprint at arXiv 2021 10.48550/arXiv.2104.10972 · DOI
  22. [26]Rasheed B.A. McLendon R.E. Herndon J.E. Friedman H.S. Friedman A.H. Bigner D.D. Bigner S.H. Alterations of the TP53 gene in human gliomas Cancer Res. 54 1994 1324 1330 8118823
  23. [27]Wick W. Weller M. van den Bent M. Sanson M. Weiler M. von Deimling A. Plass C. Hegi M. Platten M. Reifenberger G. MGMT testing—the challenges for biomarker-based glioma treatment Nat. Rev. Neurol. 10 2014 372 385 10.1038/nrneurol.2014.100 24912512 · DOI
  24. [28]Jiang S. Zanazzi G.J. Hassanpour S. Predicting prognosis and IDH mutation status for patients with lower-grade gliomas using whole slide images Sci. Rep. 11 2021 16849 10.1038/s41598-021-95948-x PMC8377095 34413349 · DOI
  25. [29]Liu S. Shah Z. Sav A. Russo C. Berkovsky S. Qian Y. Coiera E. Di Ieva A. Isocitrate dehydrogenase (IDH) status prediction in histopathology images of gliomas using deep learning Sci. Rep. 10 2020 7733 10.1038/s41598-020-64588-y 32382048 PMC7206037 · DOI
  26. [30]Yogananda C.G.B. Shah B.R. Nalawade S.S. Murugesan G.K. Yu F.F. Pinho M.C. Wagner B.C. Mickey B. Patel T.R. Fei B. MRI-based deep-learning method for determining glioma MGMT promoter methylation status AJNR. Am. J. Neuroradiol. 42 2021 845 852 10.3174/ajnr.A7029 33664111 PMC8115363 · DOI
  27. [31]Herman J.G. Graff J.R. Myöhänen S. Nelkin B.D. Baylin S.B. Methylation-specific PCR: a novel PCR assay for methylation status of CpG islands Proc. Natl. Acad. Sci. USA 93 1996 9821 9826 8790415 PMC38513
  28. [32]Fan Y. Xi L. Hughes D.S.T. Zhang J. Zhang J. Futreal P.A. Wheeler D.A. Wang W. MuSE: accounting for tumor heterogeneity using a sample-specific error model improves sensitivity and specificity in mutation calling from sequencing data Genome Biol. 17 2016 178 10.1186/s13059-016-1029-6 27557938 PMC4995747 · DOI
  29. [33]Benjamin D. Sato T. Cibulskis K. Getz G. Stewart C. Lichtenstein L. Calling somatic SNVs and indels with mutect2 Preprint at bioRxiv 2019 10.1101/861054 · DOI
  30. [34]Larson D.E. Harris C.C. Chen K. Koboldt D.C. Abbott T.E. Dooling D.J. Ley T.J. Mardis E.R. Wilson R.K. Ding L. SomaticSniper: identification of somatic point mutations in whole genome sequencing data Bioinformatics 28 2012 311 317 22155872 PMC3268238
  31. [35]Koboldt D.C. Zhang Q. Larson D.E. Shen D. McLellan M.D. Lin L. Miller C.A. Mardis E.R. Ding L. Wilson R.K. VarScan 2: somatic mutation and copy number alteration discovery in cancer by exome sequencing Genome Res. 22 2012 568 576 22300766 PMC3290792
  32. [36]Bibikova M. Le J. Barnes B. Saedinia-Melnyk S. Zhou L. Shen R. Gunderson K.L. Genome-wide DNA methylation profiling using Infinium® assay Epigenomics 1 2009 177 200 22122642
  33. [37]Du M. Liu N. Hu X. Techniques for interpretable machine learning Commun. ACM 63 2019 68 77
  34. [38]Ancona M. Ceolini E. Öztireli C. Gross M. Towards better understanding of gradient-based attribution methods for deep neural networks Preprint at arXiv 2017 10.48550/arXiv.1711.06104 · DOI
  35. [40]Sundararajan M. Taly A. Yan Q. Axiomatic Attribution for Deep Networks 2017 PMLR 3319 3328
  36. [43]Chen R.J. Krishnan R.G. Self-supervised vision transformers learn visual concepts in histopathology Preprint at arXiv 2022 10.48550/arXiv.2203.00585 · DOI · Paper page
  37. [45]Reni M. Gatta G. Mazza E. Vecht C. Crit. Rev. Oncol. Hematol. 63 2007 81 89 10.1016/j.critrevonc.2007.03.004 17482475 · DOI
  38. [47]International Agency for Research on Cancer (IARC) Section 10 meningiomas WHO Classification of Tumours of the Central Nervous System 2016 IARC Press
  39. [48]Lyndon D. Lansley J.A. Evanson J. Krishnan A.S. Dural masses: meningiomas and their mimics Insights Imaging 10 2019 11 10.1186/s13244-019-0697-7 30725238 PMC6365311 · DOI
  40. [49]Preusser M. Charles Janzer R. Felsberg J. Reifenberger G. Hamou M.F. Diserens A.C. Stupp R. Gorlia T. Marosi C. Heinzl H. Anti-O6-methylguanine-methyltransferase (MGMT) immunohistochemistry in glioblastoma multiforme: observer variability and lack of association with patient survival impede its use as clinical biomarker Brain Pathol. 18 2008 520 532 18400046 PMC8095504
  41. [50]Choi J. Lee E.Y. Shin K.-J. Minn Y.-K. Kim J. Kim S.H. IDH1 mutation analysis in low cellularity specimen: a limitation of diagnostic accuracy and a proposal for the diagnostic procedure Pathol. Res. Pract. 209 2013 284 290 23561624
  42. [51]Brown T.B. Mann B. Ryder N. Subbiah M. Kaplan J. Dhariwal P. Neelakantan A. Shyam P. Sastry G. Askell A. Language models are few-shot learners Preprint at arXiv 2020 10.48550/arXiv.2005.14165 · DOI · Paper page
  43. [52]Zhai X. Kolesnikov A. Houlsby N. Beyer L. Scaling vision transformers Preprint at arXiv 2021 10.48550/arXiv.2106.04560 · DOI
  44. [54]Simonyan K. Zisserman A. Very deep convolutional networks for large-scale image recognition Preprint at arXiv 2014 10.48550/arXiv.1409.1556 · DOI · Paper page
  45. [55]Faust K. Bala S. van Ommeren R. Portante A. Al Qawahmed R. Djuric U. Diamandis P. Intelligent feature engineering and ontological mapping of brain tumour histomorphologies by deep learning Nat. Mach. Intell. 1 2019 316 321 10.1038/s42256-019-0068-6 · DOI
  46. [57]Chen M. Zhang B. Topatana W. Cao J. Zhu H. Juengpanich S. Mao Q. Yu H. Cai X. Classification and mutation prediction based on histopathology H&E images in liver cancer using deep learning NPJ Precis. Oncol. 4 2020 14 10.1038/s41698-020-0120-3 32550270 PMC7280520 · DOI
  47. [58]Kather J.N. Pearson A.T. Halama N. Jäger D. Krause J. Loosen S.H. Marx A. Boor P. Tacke F. Neumann U.P. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer Nat. Med. 25 2019 1054 1056 10.1038/s41591-019-0462-y 31160815 PMC7423299 · DOI · Paper page
  48. [61]Goode A. Gilbert B. Harkes J. Jukic D. Satyanarayanan M. OpenSlide: a vendor-neutral software foundation for digital pathology J. Pathol. Inf. 4 2013 27 PMC3815078 24244884
  49. [62]Yu K.-H. Zhang C. Berry G.J. Altman R.B. Ré C. Rubin D.L. Snyder M. Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features Nat. Commun. 7 2016 12474 10.1038/ncomms12474 PMC4990706 27527408 · DOI · Paper page
  50. [63]Devlin J. Chang M.-W. Lee K. Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 2019 Association for Computational Linguistics 4171 4186
  51. [66]Ba J.L. Kiros J.R. Hinton G.E. Layer normalization Preprint at arXiv 2016 10.48550/arXiv.1607.06450 · DOI
  52. [67]Kraus O.Z. Ba J.L. Frey B.J. Classifying and segmenting microscopy images with deep multiple instance learning Bioinformatics 32 2016 i52 i59 10.1093/bioinformatics/btw252 27307644 PMC4908336 · DOI
  53. [68]Schrammen P.L. Ghaffari Laleh N. Echle A. Truhn D. Schulz V. Brinker T.J. Brenner H. Chang-Claude J. Alwers E. Brobeil A. Weakly supervised annotation-free cancer detection and prediction of genotype in routine histopathology J. Pathol. 256 2022 50 60 10.1002/path.5800 34561876 · DOI
  54. [73]Ying C. Cai T. Luo S. Zheng S. Ke G. He D. Shen Y. Liu T.-Y. Do transformers really perform bad for graph representation? Preprint at arXiv 2021 10.48550/arXiv.2106.05234 · DOI
  55. [76]Ishizawa K. Hirose T. Sugiyama K. Kageji T. Nobusawa S. Homma T. Komori T. Sasaki A. Pathologic diversity of glioneuronal tumor with neuropil-like islands: a histological and immunohistochemical study with a special reference to isocitrate dehydrogenase 1 (IDH1) in 5 cases Clin. Neuropathol. 31 2012 67 76 22385787
  56. [77]International Agency for Research on Cancer (IARC) Section 1 Diffuse astrocytic and oligodendroglial tumours WHO Classification of Tumours of the Central Nervous System 2016 IARC Press
  57. [79]International Agency for Research on Cancer (IARC) Section 17 Tumours of the sellar region WHO Classification of Tumours of the Central Nervous System 2016 IARC Press
  58. [80]Shrikumar A. Greenside P. Kundaje A. Learning Important Features through Propagating Activation Differences 2017 PMLR 3145 3153
  59. [82]Kokhlikyan N. Miglani V. Martin M. Wang E. Alsallakh B. Reynolds J. Melnikov A. Kliushkina N. Araya C. Yan S. Captum: a unified and generic model interpretability library for pytorch Preprint at arXiv 2020 10.48550/arXiv.2009.07896 · DOI · Paper page
  60. [83]Virtanen P. Gommers R. Oliphant T.E. Haberland M. Reddy T. Cournapeau D. Burovski E. Peterson P. Weckesser W. Bright J. SciPy 1.0: fundamental algorithms for scientific computing in Python Nat. Methods 17 2020 261 272 32015543 PMC7056644

Cited by

Sorted by citations

Authors
Li, Z., Cong, Y., Chen, X., Qi, J., Sun, J., Yan, T., Yang, H., Liu, J., Lu, E., Wang, L., Li, J., Hu, H., Zhang, C., Yang, Q., Yao, J., Yao, P., Jiang, Q., Liu, W., Song, J., Carin, L., Chen, Y., Zhao, S., & Gao, X.
Affiliations
  • King Abdullah University of Science and Technology (SA)
  • First Affiliated Hospital of Harbin Medical University (CN)
  • Harbin Medical University (CN)
  • Suffolk University (US)
  • Monash University (AU)
  • Southern University of Science and Technology (CN)
Corresponding author
  • Qi, Jiping
  • Carin, Lawrence
  • Chen, Yupeng
  • Zhao, Shiguang
  • Gao, Xin
ORCID
Venue
iScience
Publisher
Elsevier BV
Published
2023-01
Volume
26
Issue
1
Pages
105872
ISSN
2589-0042 (ISSN-L)
DOI
10.1016/j.isci.2022.105872
PMC
PMC9839963
OpenAlex
W4312204014
Cited by
82
References
10
Citations by year
2026: 22 · 2025: 25 · 2024: 21 · 2023: 14
FWCI
4.94
Type
journal article
Language
en
Open access
Licence
Locations
  • iScience (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • King Abdullah University of Science and Technology Repository (King Abdullah University of Science and Technology) (submittedVersion, cc-by-nc-nd)
Funders
King Abdullah University of Science and Technology
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Life Sciences › Neuroscience › Neurology › Brain Tumor Detection and Classification
Keywords
vision transformer, weakly supervised learning, histopathology image analysis, primary brain tumors, H&E-stained images, DNA methylation, tumor classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review