Handcrafted Histological Transformer (H2T): Unsupervised representation of whole slide images
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Diagnostic, prognostic and therapeutic decision-making of cancer in pathology clinics can now be carried out based on analysis of multi-gigapixel tissue images, also known as whole-slide images (WSIs). Recently, deep convolutional neural networks (CNNs) have been proposed to derive unsupervised WSI representations; these are attractive as they rely less on expert annotation which is cumbersome. However, a major trade-off is that higher predictive power generally comes at the cost of interpretability, posing a challenge to their clinical use where transparency in decision-making is generally expected. To address this challenge, we present a handcrafted framework based on deep CNN for constructing holistic WSI-level representations. Building on recent findings about the internal working of the Transformer in the domain of natural language processing, we break down its processes and handcraft them into a more transparent framework that we term as the Handcrafted Histological Transformer or H2T. Based on our experiments involving various datasets consisting of a total of 10,042 WSIs, the results demonstrate that H2T based holistic WSI-level representations offer competitive performance compared to recent state-of-the-art methods and can be readily utilized for various downstream analysis tasks. Finally, our results demonstrate that the H2T framework can be up to 14 times faster than the Transformer models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Solving the multiple instance problem with axis-parallel rectangles
Dietterich, T. G., Lathrop, R. H., & Lozano-Pérez, T. (1997) · Artificial Intelligence · doi:10.1016/s0004-3702(96)00034-3 · Cited by 2,866
Clinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020) · arXiv · doi:10.48550/arxiv.2006.09882 · Cited by 2,499
Data-efficient and weakly supervised computational pathology on whole-slide images
Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Chen, R. J., Barbieri, M., & Mahmood, F. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-020-00682-w · Cited by 2,282
Histopathological Image Analysis: A Review
Gurcan, M. N., Boucheron, L. E., Can, A., Madabhushi, A., Rajpoot, N. M., & Yener, B. (2009) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2009.2034865 · Cited by 2,052
Self-citationHendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01501 · Cited by 1,090
Kather, J. N., Krisam, J., Charoentong, P., Luedde, T., Herpel, E., Weis, C.-A., Gaiser, T., Marx, A., Valous, N. A., Ferber, D., et al. (2019) · PLOS Medicine · doi:10.1371/journal.pmed.1002730 · Cited by 1,082
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2019) · arXiv · doi:10.48550/arxiv.1907.07174 · Cited by 1,058
Research works printed but not linked: 33
- Abbet, C., Zlobec, I., Bozorgtabar, B., Thiran, J.P., 2020. Divide-and-Rule: Self-Supervised Learning for Survival Analysis in Colorectal Cancer. In: MICCAI. · DOI
- Abels, 2019, Computational pathology definitions, best practices, and recommendations for regulatory guidance: A white paper from the digital pathology association, J. Pathol., 249, 10.1002/path.5331 · DOI · Paper page
- Andrews, S., Tsochantaridis, I., Hofmann, T., 2002. Support Vector Machines for Multiple-Instance Learning. In: NIPS, Vol. 2, No. 3. p. 7.
- Awan, 2017, Glandular morphometrics for objective grading of colorectal adenocarcinoma histology images, Sci. Rep., 7, 16852, 10.1038/s41598-017-16516-w · DOI · Paper page
- Azam, 2021, Diagnostic concordance and discordance in digital pathology: a systematic review and meta-analysis, J. Clin. Pathol., 74, 448, 10.1136/jclinpath-2020-206764 · DOI · Paper page
- Cao, 2016, Clinical impacts of a micropapillary pattern in lung adenocarcinoma: a review, OncoTargets Ther., 9, 149
- Chen, 2022, Deep learning for instance retrieval: A survey, IEEE Trans. Pattern Anal. Mach. Intell., 1
- Diao, 2021, Human-interpretable image features derived from densely mapped cancer pathology slides predict diverse molecular phenotypes, Nature Commun., 12, 1613, 10.1038/s41467-021-21896-9 · DOI · Paper page
- Doersch, C., Zisserman, A., 2017. Multi-task self-supervised visual learning. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2051–2060. · DOI · Paper page
- Elston, 1991, Pathological prognostic factors in breast cancer. I. The value of histological grade in breast cancer: experience from a large study with long-term follow-up, Histopathology, 19, 403, 10.1111/j.1365-2559.1991.tb00229.x · DOI · Paper page
- Gentles, 2015, The prognostic landscape of genes and infiltrating immune cells across human cancers, Nature Med., 21, 10.1038/nm.3909 · DOI · Paper page
- Goldstraw, 2016, The IASLC lung cancer staging project: Proposals for revision of the TNM stage groupings in the forthcoming (eighth) edition of the TNM classification for lung cancer, J. Thorac. Oncol., 11, 39, 10.1016/j.jtho.2015.09.009 · DOI · Paper page
- Graham, 2020, Dense steerable filter cnns for exploiting rotational symmetry in histology images, IEEE Trans. Med. Imaging, 39, 4124, 10.1109/TMI.2020.3013246 · DOI
- Graham, 2019, Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images, Med. Image Anal., 10.1016/j.media.2019.101563 · DOI · Paper page
- Hamilton, 1994, Expert system support using Bayesian belief networks in the diagnosis of fine needle aspiration biopsy specimens of the breast, J. Clin. Pathol., 47, 329, 10.1136/jcp.47.4.329 · DOI
- Kandemir, 2015, Computer-aided diagnosis from weak supervision: A benchmarking study, Comput. Med. Imaging Graph., 42, 44, 10.1016/j.compmedimag.2014.11.010 · DOI
- Keenan, 2000, An automated machine vision system for the histological grading of cervical intraepithelial neoplasia (CIN), J. Pathol., 192, 351, 10.1002/1096-9896(2000)9999:9999<::AID-PATH708>3.0.CO;2-I · doi:10.1002/1096-9896(2000)9999:9999<::aid-path708>3.0.co;2-i
- Kim, D., Cho, D., Kweon, I.S., 2019. Self-supervised video representation learning with space-time cubic puzzles. In: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, No. 01. pp. 8545–8552. · DOI · Paper page
- Koohbanani, 2021, Self-path: Self-supervision for classification of pathology images with limited annotations, IEEE Trans. Med. Imaging, 40, 2845, 10.1109/TMI.2021.3056023 · DOI · Paper page
- Kromp, 2020, An annotated fluorescence image dataset for training nuclear segmentation methods, Sci. Data, 7, 1, 10.1038/s41597-020-00608-w · DOI
- Kwak, 2017, Multiview boosting digital pathology analysis of prostate cancer, Comput. Methods Programs Biomed., 142, 91, 10.1016/j.cmpb.2017.02.023 · DOI
- Li, 2020, Deep learning methods for lung cancer segmentation in whole-slide histopathology images-the acdc@ lunghp challenge 2019, IEEE J. Biomed. Health Inf.
- Liu, 2012, Isolation-based anomaly detection, ACM Trans. Knowl. Discov. Data (TKDD), 6, 1, 10.1145/2133360.2133363 · DOI · Paper page
- Sertel, 2009, Computer-aided prognosis of neuroblastoma on whole-slide images: Classification of stromal development, Pattern Recognit., 42, 1093, 10.1016/j.patcog.2008.08.027 · DOI · Paper page
- Shaban, 2019, A novel digital score for abundance of tumour infiltrating lymphocytes predicts disease free survival in oral squamous cell carcinoma, Sci. Rep., 9, 1, 10.1038/s41598-019-49710-z · DOI · Paper page
- Shield, 2017, The global incidence of lip, oral cavity, and pharyngeal cancers by subsite in 2012, CA: Cancer J. Clin., 67, 51
- Solis, 2012, Histologic patterns and molecular characteristics of lung adenocarcinoma associated with clinical outcome, Cancer, 118, 2889, 10.1002/cncr.26584 · DOI
- Tabesh, 2007, Multifeature prostate cancer diagnosis and gleason grading of histological images, IEEE Trans. Med. Imaging, 26, 1366, 10.1109/TMI.2007.898536 · DOI · Paper page
- Tan, F., Yuan, J., Ordonez, V., 2021. Instance-level image retrieval using reranking transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12105–12115. · DOI
- Van der Maaten, 2008, Visualizing data using t-sne, J. Mach. Learn. Res., 9
- Verma, 2021, MoNuSAC2020: A multi-organ nuclei segmentation and classification challenge, IEEE Trans. Med. Imaging, 40, 3413, 10.1109/TMI.2021.3085712 · DOI · Paper page
- Vu, 2020, Unsupervised tumor characterization via conditional generative adversarial networks, IEEE J. Biomed. Health Inf., 25, 348, 10.1109/JBHI.2020.2993560 · DOI
- Wahab, 2021, Semantic annotation for computational pathology: Multidisciplinary experience and best practice recommendations, J. Pathol. Clin. Res.
Cited by
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560
Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images
Lu, M. Y., Chen, B., Zhang, A., Williamson, D. F. K., Chen, R. J., Ding, T., Le, L. P., Chuang, Y.-S., & Mahmood, F. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01893 · Cited by 122
Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology
Song, A. H., Chen, R. J., Ding, T., Williamson, D. F. K., Jaume, G., & Mahmood, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01099 · Cited by 79
Multistain Pretraining for Slide Representation Learning in Pathology
Jaume, G., Vaidya, A., Zhang, A., H. Song, A., J. Chen, R., Sahai, S., Mo, D., Madrigal, E., Phi Le, L., & Mahmood, F. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73414-4_2 · Cited by 27
Metadata
- Authors
- Vu, Q. D., Rajpoot, K., Raza, S. E. A., & Rajpoot, N.
- Venue
- Medical Image Analysis
- Publisher
- Elsevier BV
- Published
- 2023-04
- Volume
- 85
- Pages
- 102743
- ISSN
- 1361-8415 (ISSN-L)
- OpenAlex
- W4317473799
- Cited by
- 55
- References
- 19
- FWCI
- 4.83
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Funders
- Medical Research Council (MR/P015476/1)
- Royal Marsden NHS Foundation Trust
- Royal Society
- Alan Turing Institute
- European Research Council (H2020 IMI2-RIA)
- UK Research and Innovation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Handcrafted Histological Transformer (H2T): Unsupervised representation of whole slide images. https://app.cochat.ai/papers/10.1016/j.media.2023.102743
- The paper
- Vu, Q. D., Rajpoot, K., Raza, S. E. A., & Rajpoot, N. (2023). Handcrafted Histological Transformer (H2T): Unsupervised representation of whole slide images. Medical Image Analysis. https://doi.org/10.1016/j.media.2023.102743
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.