Multi-scale Domain-adversarial Multiple-instance CNN for Cancer Subtype Classification with Unannotated Histopathological Images
Open accessThe DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
Abstract
We propose a new method for cancer subtype classification from histopathological images, which can automatically detect tumor-specific features in a given whole slide image (WSI). The cancer subtype should be classified by referring to a WSI, i.e., a large-sized image (typically 40,000x40,000 pixels) of an entire pathological tissue slide, which consists of cancer and non-cancer portions. One difficulty arises from the high cost associated with annotating tumor regions in WSIs. Furthermore, both global and local image features must be extracted from the WSI by changing the magnifications of the image. In addition, the image features should be stably detected against the differences of staining conditions among the hospitals/specimens. In this paper, we develop a new CNN-based cancer subtype classification method by effectively combining multiple-instance, domain adversarial, and multi-scale learning frameworks in order to overcome these practical difficulties. When the proposed method was applied to malignant lymphoma subtype classifications of 196 cases collected from multiple hospitals, the classification performance was significantly better than the standard CNN or other conventional methods, and the accuracy compared favorably with that of standard pathologists.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 12×
“Feature extractor G f : x 7→ h is a CNN which maps a 224 × 224 -pixel image x into a Q -dimensional feature vector h .”
BackgroundInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 2×
“In this experiment, VGG16 [31] pre-trained with ImageNet was employed as the feature extractor Gf(·; θf) and the dimension of the output features was Q = 25, 088.”
BackgroundMethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 12×
“Feature extractor G f : x 7→ h is a CNN which maps a 224 × 224 -pixel image x into a Q -dimensional feature vector h .”
BackgroundInfluentialXception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · cited 12×
“Feature extractor G f : x 7→ h is a CNN which maps a 224 × 224 -pixel image x into a Q -dimensional feature vector h .”
BackgroundInfluentialDomain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318 · cited 2×
“Domain-adversarial (DA) [15] training has been proposed to ignore the differences among training instances that do not contribute to the classification task.”
BackgroundMethodEhteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293 · cited 1×
“Since a whole slide image (WSI) obtained by digitizing an entire pathological tissue slide is too large to feed into a CNN [22, 31, 18, 8], it is common to extract a large number of patches from the WSI [9, 12, 27, 37, 19, 34, 5, 38, 2, 17].”
BackgroundReinhard, E., Adhikhmin, M., Gooch, B., & Shirley, P. (2001) · IEEE Computer Graphics and Applications · doi:10.1109/38.946629 · Cited by 2,937 · cited 1×
“To overcome this difficulty, appropriate pre-processing such as color normalization [30, 25, 21, 4, 39] or color augmentation [23, 29] would be required.”
BackgroundSolving the multiple instance problem with axis-parallel rectangles
Dietterich, T. G., Lathrop, R. H., & Lozano-Pérez, T. (1997) · Artificial Intelligence · doi:10.1016/s0004-3702(96)00034-3 · Cited by 2,866 · cited 1×
“Various models and learning algorithms for MIL have been studied in the literatures [14, 26, 40, 1, 24, 7, 36].”
BackgroundClinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799 · cited 1×
“MIL has been successfully applied to classification problems with histopathological images [10, 13, 20, 11, 32, 6].”
MethodOn the momentum term in gradient descent learning algorithms
Qian, N. (1999) · Neural Networks · doi:10.1016/s0893-6080(98)00116-6 · Cited by 2,312 · cited 1×
“Parameters in the network were optimized by SGD momentum [28], where the learning rate and momentum were set to 0.”
MethodMitosis Detection in Breast Cancer Histology Images with Deep Neural Networks
Cireşan, D. C., Giusti, A., Gambardella, L. M., & Schmidhuber, J. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_51 · Cited by 1,536 · cited 1×
“Since a whole slide image (WSI) obtained by digitizing an entire pathological tissue slide is too large to feed into a CNN [22, 31, 18, 8], it is common to extract a large number of patches from the WSI [9, 12, 27, 37, 19, 34, 5, 38, 2, 17].”
BackgroundA method for normalizing histology slides for quantitative analysis
Macenko, M., Niethammer, M., Marron, J. S., Borland, D., Woosley, J. T., Guan, X., Schmitt, C., & Thomas, N. E. (2009) · 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro · doi:10.1109/isbi.2009.5193250 · Cited by 1,482 · cited 1×
“To overcome this difficulty, appropriate pre-processing such as color normalization [30, 25, 21, 4, 39] or color augmentation [23, 29] would be required.”
BackgroundSupport Vector Machines for Multiple-Instance Learning
Andrews, S. J., Tsochantaridis, I., & Hofmann, T. (2002) · neural information processing systems · Cited by 1,401
A Framework for Multiple-Instance Learning
Maron, O., & Lozano‐Pérez, T. (1997) · neural information processing systems · Cited by 1,230
Patch-Based Convolutional Neural Network for Whole Slide Tissue Image Classification
Hou, L., Samaras, D., Kurc, T. M., Gao, Y., Davis, J. E., & Saltz, J. H. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.266 · Cited by 866 · cited 1×
“Since a whole slide image (WSI) obtained by digitizing an entire pathological tissue slide is too large to feed into a CNN [22, 31, 18, 8], it is common to extract a large number of patches from the WSI [9, 12, 27, 37, 19, 34, 5, 38, 2, 17].”
BackgroundDeep Learning for Identifying Metastatic Breast Cancer
Wang, D., Khosla, A., Gargeya, R., Irshad, H., & Beck, A. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05718 · Cited by 756 · cited 1×
“Since a whole slide image (WSI) obtained by digitizing an entire pathological tissue slide is too large to feed into a CNN [22, 31, 18, 8], it is common to extract a large number of patches from the WSI [9, 12, 27, 37, 19, 34, 5, 38, 2, 17].”
BackgroundAttention-based Deep Multiple Instance Learning
Ilse, M., Tomczak, J. M., & Welling, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.04712 · Cited by 636
Cruz-Roa, A., Basavanhally, A., González, F., Gilmore, H., Feldman, M., Ganesan, S., Shih, N., Tomaszewski, J., & Madabhushi, A. (2014) · SPIE Proceedings · doi:10.1117/12.2043872 · Cited by 573
Bandi, P., Geessink, O., Manson, Q., Van Dijk, M., Balkenhol, M., Hermsen, M., Ehteshami Bejnordi, B., Lee, B., Paeng, K., Zhong, A., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2867350 · Cited by 569 · cited 1×
“Since a whole slide image (WSI) obtained by digitizing an entire pathological tissue slide is too large to feed into a CNN [22, 31, 18, 8], it is common to extract a large number of patches from the WSI [9, 12, 27, 37, 19, 34, 5, 38, 2, 17].”
BackgroundKhan, A. M., Rajpoot, N., Treanor, D., & Magee, D. (2014) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2303294 · Cited by 518 · cited 1×
“To overcome this difficulty, appropriate pre-processing such as color normalization [30, 25, 21, 4, 39] or color augmentation [23, 29] would be required.”
BackgroundXu, Y., Jia, Z., Wang, L.-B., Ai, Y., Zhang, F., Lai, M., & Chang, E. I.-C. (2017) · BMC Bioinformatics · doi:10.1186/s12859-017-1685-x · Cited by 416 · cited 1×
“Since a whole slide image (WSI) obtained by digitizing an entire pathological tissue slide is too large to feed into a CNN [22, 31, 18, 8], it is common to extract a large number of patches from the WSI [9, 12, 27, 37, 19, 34, 5, 38, 2, 17].”
BackgroundDeep multiple instance learning for image classification and auto-annotation
Wu, J., Yu, Y., Huang, C., & Yu, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298968 · Cited by 386 · cited 1×
“Various models and learning algorithms for MIL have been studied in the literatures [14, 26, 40, 1, 24, 7, 36].”
BackgroundMultiple instance learning for histopathological breast cancer image classification
Sudharshan, P. J., Petitjean, C., Spanhol, F., Oliveira, L. E., Heutte, L., & Honeine, P. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.09.049 · Cited by 369
Stain Specific Standardization of Whole-Slide Histopathological Images
Ehteshami Bejnordi, B., Litjens, G., Timofeeva, N., Otte-Holler, I., Homeyer, A., Karssemeijer, N., & van der Laak, J. A. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2476509 · Cited by 290 · cited 1×
“To overcome this difficulty, appropriate pre-processing such as color normalization [30, 25, 21, 4, 39] or color augmentation [23, 29] would be required.”
BackgroundDeep Convolutional Neural Networks for Breast Cancer Histology Image Analysis
Rakhlin, A., Shvets, A., Iglovikov, V., & Kalinin, A. A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93000-8_83 · Cited by 219
Xu, Y., Jia, Z., Ai, Y., Zhang, F., Lai, M., & Chang, E. I.-C. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178109 · Cited by 187 · cited 1×
“Since a whole slide image (WSI) obtained by digitizing an entire pathological tissue slide is too large to feed into a CNN [22, 31, 18, 8], it is common to extract a large number of patches from the WSI [9, 12, 27, 37, 19, 34, 5, 38, 2, 17].”
BackgroundDomain-Adversarial Neural Networks to Address the Appearance Variability of Histopathology Images
Lafarge, M. W., Pluim, J. P. W., Eppenhof, K. A. J., Moeskops, P., & Veta, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67558-9_10 · Cited by 179
Neural Networks for Multi-Instance Learning
Zhou, Z., & Zhang, M.-L. (2002) · Cited by 146
Adaptive Weighting Multi-Field-Of-View CNN for Semantic Segmentation in Pathology
Tokunaga, H., Teramoto, Y., Yoshizawa, A., & Bise, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01288 · Cited by 141 · cited 2×
“Although most of the existing studies on histopathological image analysis use a fixed single scale, some studies use multiple scales [3, 16, 35, 33].”
BackgroundMethodDas, K., Conjeti, S., Roy, A. G., Chatterjee, J., & Sheet, D. (2018) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2018.8363642 · Cited by 90 · cited 1×
“MIL has been successfully applied to classification problems with histopathological images [10, 13, 20, 11, 32, 6].”
MethodMultiple Instance Learning for Heterogeneous Images: Training a CNN for Histopathology
Couture, H. D., Marron, J. S., Perou, C. M., Troester, M. A., & Niethammer, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00934-2_29 · Cited by 82
Automated discrimination of lower and higher grade gliomas based on histopathological image analysis
Mousavi, H. S., Monga, V., Rao, G., & Rao, A. U. K. (2015) · Journal of Pathology Informatics · doi:10.4103/2153-3539.153914 · Cited by 81 · cited 1×
“Since a whole slide image (WSI) obtained by digitizing an entire pathological tissue slide is too large to feed into a CNN [22, 31, 18, 8], it is common to extract a large number of patches from the WSI [9, 12, 27, 37, 19, 34, 5, 38, 2, 17].”
BackgroundMulti-instance multi-label image classification: A neural approach
Chen, Z., Chi, Z., Fu, H. C., & Feng, D. D. (2012) · Neurocomputing · doi:10.1016/j.neucom.2012.08.001 · Cited by 77 · cited 1×
“Various models and learning algorithms for MIL have been studied in the literature [15, 29, 43, 1, 27, 7, 39].”
BackgroundBejnordi, B., Litjens, G., Hermsen, M., Karssemeijer, N., & van der Laak, J. A. (2015) · Medical Imaging · doi:10.1117/12.2081768 · Cited by 54
Classification of lung cancer histology images using patch-level summary statistics
Khurram, S. A., Graham, S., Shaban, M., Qaiser, T., & Rajpoot, N. (2018) · SPIE Medical Imaging: Digital and Computational Pathology · doi:10.1117/12.2293855 · Cited by 46
Histopathology stain-color normalization using deep generative models
Zanjani, F. G., Zinger, S., Bejnordi, B. E., & van der Laak, J. A. W. M. (2018) · Cited by 44
Cosatto, E., Laquerre, P.-F., Malon, C. D., Graf, H.-P., Saito, A., Kiyuna, T., Marugame, A., & Kamijo, K. (2013) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2007047 · Cited by 44 · cited 1×
“MIL has been successfully applied to classification problems with histopathological images [10, 13, 20, 11, 32, 6].”
MethodMulti-scale learning based segmentation of glands in digital colonrectal pathology images
Gao, Y., Liu, W., Arjun, S., Zhu, L., Ratner, V., Kurç, T., Saltz, J. H., & Tannenbaum, A. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2216790 · Cited by 18
Wetteland, R., Engan, K., Eftestøl, T. C., Kvikstad, V., & Janssen, E. A. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01178 · Cited by 10
Cited by
Li, B., Li, Y., & Eliceiri, K. W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01409 · Cited by 957
“2791 MS-ABMIL [17] Multiple 0.”
BackgroundMethodInfluentialScaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning
Chen, R. J., Chen, C., Li, Y., Chen, T. Y., Trister, A. D., Krishnan, R. G., & Mahmood, F. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01567 · Cited by 621
“The integration of magnification objectives within WSIs has been followed in other works as well [29,32,56,58], however, we note that combining visual tokens across objectives would not share the same scale.”
BackgroundTakahashi, S., Sakaguchi, Y., Kouno, N., Takasawa, K., Ishizu, K., Akagi, Y., Aoyama, R., Teraya, N., Bolatkan, A., Shinkai, N., et al. (2024) · Journal of Medical Systems · doi:10.1007/s10916-024-02105-8 · Cited by 343
Cross-scale multi-instance learning for pathological image diagnosis
Deng, R., Cui, C., Remedios, L. W., Bao, S., Womick, R. M., Chiron, S., Li, J., Roland, J. T., Lau, K. S., Liu, Q., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103124 · Cited by 95
“…al., (Hou et al., 2016) proposed a decision fusion model that aggregated patch-level predictions generated by patch-level CNNs. Hashimoto et al., (Hashimoto et al., 2020b) proposed a novel CNN-based technique for cancer subtype classification by effectively merging multiple-instance, domain…”
BackgroundMethodInfluentialMorphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology
Song, A. H., Chen, R. J., Ding, T., Williamson, D. F. K., Jaume, G., & Mahmood, F. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01099 · Cited by 79
Metadata
- Authors
- Hashimoto, N., Fukushima, D., Koga, R., Takagi, Y., Ko, K., Kohno, K., Nakaguro, M., Nakamura, S., Hontani, H., & Takeuchi, I.
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 3851-3860
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3034447539
- Cited by
- 227
- References
- 39
- FWCI
- 11.69
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.