Locality Sensitive Deep Learning for Detection and Classification of Nuclei in Routine Colon Cancer Histology Images
The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Detection and classification of cell nuclei in histopathology images of cancerous tissue stained with the standard hematoxylin and eosin stain is a challenging task due to cellular heterogeneity. Deep learning approaches have been shown to produce encouraging results on histopathology images in various studies. In this paper, we propose a Spatially Constrained Convolutional Neural Network (SC-CNN) to perform nucleus detection. SC-CNN regresses the likelihood of a pixel being the center of a nucleus, where high probability values are spatially constrained to locate in the vicinity of the centers of nuclei. For classification of nuclei, we propose a novel Neighboring Ensemble Predictor (NEP) coupled with CNN to more accurately predict the class label of detected cell nuclei. The proposed approaches for detection and classification do not require segmentation of nuclei. We have evaluated them on a large dataset of colorectal adenocarcinoma images, consisting of more than 20,000 annotated nuclei belonging to four different classes. Our results show that the joint detection and classification of the proposed SC-CNN and NEP produces the highest average F1 score as compared to other recently published approaches. Prospectively, the proposed methods could offer benefit to pathology practice in terms of quantitative analysis of tissue constituents in whole-slide images, and potentially lead to a better understanding of cancer.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Sirinukunwattana et al. [292] utilized 100 histology images of colorectal adenocarcinoma to detect cell nuclei using CNNs.”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [292]
The cited source says what the claim attributes to it.
In the citing paper
“This CNN detects the center of nuclei using the surrounding spatial context and spatial regression.”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [292]
In the citing paper
“Some recent studies have shown that deep learning algorithms are successfully used for medical image segmentation [58], computer aided diagnosis [59,60,61], disease detection and classification [62,63,64,65] and medical image retrieval [66,67].”
From Medical image analysis using convolutional neural networks: a review (2018) · cites this paper as [65]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 2×
“In both networks, a rectified linear unit (ReLU) activation function [32] is used after each convolution layer and the first two fully-connected layers (1st, 3rd, 5th and 6th layer).”
MethodU-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
(2014) · Cited by 33,419
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066 · cited 2×
“…(i.e., weight vectors and bias values) in the network, we solve (2) using the following loss function: (10) where is the cross-entropy loss [29]–[31] defined by (11) and is a small constant, which is set to be the ratio of the total number of non-zero probability pixels and zero probability…”
Method(2012) · Cited by 9,062
U-Net: Convolutional Networks for Biomedical Image SegmentationPreprint
Ronneberger, O., Fischer, P., & Brox, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.04597 · Cited by 5,282 · cited 1×
“The use of small patches not only increases the amount of training data which is crucial for CNNs, but essentially also localizes our analysis to small nuclei in images [10], [11].”
BackgroundA human colon cancer cell capable of initiating tumour growth in immunodeficient mice
O’Brien, C. A., Pollett, A., Gallinger, S., & Dick, J. E. (2007) · Nature · doi:10.1038/nature05372 · Cited by 4,087 · cited 1×
“other factors involved in tumor development [1], [2].”
BackgroundPattern recognition and machine learning
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5091 · Cited by 2,650
(2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-24553-9 · Cited by 2,285
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems
Hand, D. J., & Till, R. J. (2001) · Machine Learning · doi:10.1023/a:1010920819831 · Cited by 2,276 · cited 1×
“We also considered an area under the receiver operating characteristic curve for multiclass classi-fication (multiclass AUC) [36].”
MethodMitosis Detection in Breast Cancer Histology Images with Deep Neural Networks
Cireşan, D. C., Giusti, A., Gambardella, L. M., & Schmidhuber, J. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_51 · Cited by 1,536 · cited 2×
“As opposed to other approaches [7], [8] that do not enforce the pixels close to the center of a nucleus to have a higher probability value than those further away, the predicted probability values produced by SC-CNN are topologically constrained such that high probability values are concentrated in the vicinity of the…”
MethodResult(2012) · neural information processing systems · Cited by 1,291
Exploring Strategies for Training Deep Neural Networks
Larochelle, H., Bengio, Y., Louradour, J., & Lamblin, P. (2009) · doi:10.1145/1577069.1577070 · Cited by 1,095
Stacked Sparse Autoencoder (SSAE) for Nuclei Detection on Breast Cancer Histopathology Images
Xu, J., Xiang, L., Liu, Q., Gilmore, H., Wu, J., Tang, J., & Madabhushi, A. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2458702 · Cited by 854 · cited 3×
“As opposed to other approaches [7], [8] that do not enforce the pixels close to the center of a nucleus to have a higher probability value than those further away, the predicted probability values produced by SC-CNN are topologically constrained such that high probability values are concentrated in…”
BackgroundMethodResultSingle-cell dissection of transcriptional heterogeneity in human colon tumors
Dalerba, P., Kalisky, T., Sahoo, D., Rajendran, P. S., Rothenberg, M. E., Leyrat, A. A., Sim, S., Okamoto, J., Johnston, D. M., Qian, D., et al. (2011) · Nature Biotechnology · doi:10.1038/nbt.2038 · Cited by 703 · cited 1×
“other factors involved in tumor development [1], [2].”
BackgroundBreast Cancer Histopathology Image Analysis: A Review
Veta, M., Pluim, J. P. W., van Diest, P. J., & Viergever, M. A. (2014) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2303852 · Cited by 696 · cited 1×
“Recently proposed techniques for nucleus detection in routine H&E histology images rely on morphological features such as symmetry and stability of the nuclear region to identify nuclei [12].”
Improved Automatic Detection and Segmentation of Cell Nuclei in Histopathology Images
Al-Kofahi, Y., Lassoued, W., Lee, W., & Roysam, B. (2010) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2009.2035102 · Cited by 681 · cited 1×
“Al Kofahi et al. [14] proposed a graph cut-based method that is initialized using response of the image to Laplacian of Gaussian (LoG) filter.”
MethodIrshad, H., Veillard, A., Roux, L., & Racoceanu, D. (2014) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2013.2295804 · Cited by 637 · cited 1×
“An alternative and more efficient approach which is taken in this paper is to use morphological clues in local neighborhoods to develop automated cellular recognition via image analysis which can then be deployed for sophisticated tissue morphometry in future [6].”
MethodMatConvNet - Convolutional Neural Networks for MATLAB
Vedaldi, A., & Lenc, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.4564 · Cited by 578 · cited 2×
“We implemented both networks using MatConvNet [34].”
Khan, A. M., Rajpoot, N., Treanor, D., & Magee, D. (2014) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2303294 · Cited by 518 · cited 1×
“The hematoxylin intensity was obtained using a recently proposed color deconvolution method [35].”
MethodSelf-citationQuantitative Image Analysis of Cellular Heterogeneity in Breast Tumors Complements Genomic Profiling
Yuan, Y., Failmezger, H., Rueda, O. M., Ali, H. R., Gräf, S., Chin, S.-F., Schwarz, R. F., Curtis, C., Dunning, M. J., Bardwell, H., et al. (2012) · Science Translational Medicine · doi:10.1126/scitranslmed.3004330 · Cited by 426
Cruz-Roa, A. A., Arevalo Ovalle, J. E., Madabhushi, A., & González Osorio, F. A. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_50 · Cited by 403 · cited 1×
“Cruz et al. [26] showed that for detection of basal-cell carcinoma, features learned using deep learning approaches produce superior and stable results compared to pre-defined bag of feature and canonical (discrete cosine transform/wavelet transform based) representations.”
ResultAutomatic Nuclei Segmentation in H&E Stained Breast Cancer Histopathology Images
Veta, M., van Diest, P. J., Kornegoor, R., Huisman, A., Viergever, M. A., & Pluim, J. P. W. (2013) · PLoS ONE · doi:10.1371/journal.pone.0070221 · Cited by 369 · cited 1×
“Similarly, Veta et al. [16] relied on the direction of gradient to identify the centers of nuclei.”
Backgroundvan Muijen, G. N. P., Ruiter, D. J., Franke, W. W., Achtstätter, T., Haasnoot, W. H. B., Ponec, M., & Warnaar, S. O. (1986) · Experimental Cell Research · doi:10.1016/0014-4827(86)90429-5 · Cited by 306 · cited 1×
“That will, however, require deep biological understanding of tumors to identify informative markers and is costly in terms of laboratory time and the use of tissue to study cell types [5].”
BackgroundTumor budding in colorectal carcinoma: time to take notice
Mitrovic, B., Schaeffer, D. F., Riddell, R. H., & Kirsch, R. (2012) · Modern Pathology · doi:10.1038/modpathol.2012.94 · Cited by 289 · cited 1×
“…should be pointed out that this type of correction should be used with caution, as it may falsely eliminate biologically important phenomena such as tumor budding which consists of a small number of tumor nuclei [38] or isolated islands of tumor nuclei appearing at the invasive front of the tumor.”
BackgroundAli, S., & Madabhushi, A. (2012) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2012.2190089 · Cited by 243 · cited 1×
“Ali et al. [18] proposed an active contour-based approach to detect and segment overlapping nuclei based on shape models, which is highly variable in the case of tumor nuclei.”
BackgroundXie, Y., Xing, F., Kong, X., Su, H., & Yang, L. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-24574-4_43 · Cited by 193 · cited 3×
“[27] recently proposed structural regression CNN capable of learning a proximity map of cell nuclei and was shown by the authors to provide more accurate detection results.”
MethodClassification of mitotic figures with convolutional neural networks and seeded blob features
Malon, C. D., & Cosatto, E. (2013) · Journal of Pathology Informatics · doi:10.4103/2153-3539.112694 · Cited by 192 · cited 1×
“Malon et al. [21] trained CNN classifier to classify mitotic and non-mitotic cells using color, texture and shape information.”
BackgroundLearning to Detect Cells Using Non-overlapping Extremal Regions
Arteta, C., Lempitsky, V., Noble, J. A., & Zisserman, A. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33415-3_43 · Cited by 177 · cited 10×
“The better performance of NEP than that of SSPP allows us to hypothesize that NEP is more resilient to variability in the appearance of nuclei.”
BackgroundMethodInfluentialGrading Nuclear Pleomorphism on Histological Micrographs
Cosatto, E., Miller, M. L., Graf, H. P., & Meyer, J. S. (2008) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2008.4761112 · Cited by 151 · cited 2×
“Cosatto et al. [13] proposed detection of cell nuclei using difference of Gaussian (DoG) followed by Hough transform to find radially symmetrical shapes.”
BackgroundMethodEfficient nucleus detector in histopathology images
Vink, J. P., van Leeuwen, M. B., van Deurzen, C. H. M., & de Haan, G. (2012) · Journal of Microscopy · doi:10.1111/jmi.12001 · Cited by 117 · cited 1×
“In a recent study, Vink et al. [19] employed Ad-aBoost classifier to train two detectors—one using pixel-based features and the other based on Haar-like features—and merged the results of two detectors to detect the nuclei in immunohistochemistry stained breast tissue images.”
BackgroundCascaded ensemble of convolutional neural networks and handcrafted features for mitosis detection
Wang, H., Roa, Á. A. C., Basavanhally, A. N., Gilmore, H. L., Shih, N. N. C., Feldman, M., Tomaszewski, J., Gonzalez, F. A., & Madabhushi, A. (2014) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2043902 · Cited by 115 · cited 1×
“Wang et al. [25] proposed a cascaded classifier which uses a combination of hand-crafted features and features learned through CNN to detect mitotic cells.”
MethodDeep Voting: A Robust Approach Toward Nucleus Localization in Microscopy Images
Xie, Y., Kong, X., Xing, F., Liu, F., Su, H., & Yang, L. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_45 · Cited by 91 · cited 4×
“However, SC-CNN generates a probability mask for spatial regression based on these parameters rather than employing them directly for detection as done by Xie et al. [28].”
BackgroundMethodInfluentialLewis, J. S., Ali, S., Luo, J. R., Thorstad, W. L., & Madabhushi, A. (2013) · The American Journal of Surgical Pathology · doi:10.1097/pas.0000000000000086 · Cited by 86 · cited 1×
“The spatial arrangement of these heterogeneous cell types has also been shown to be related to cancer grades [3], [4].”
BackgroundProstate cancer detection: Fusion of cytological and textural features
Nguyen, K., Jain, A. K., & Sabata, B. (2012) · Journal of Pathology Informatics · doi:10.4103/2153-3539.92030 · Cited by 73 · cited 1×
“Nguyen et al. [22] classified nuclei on the basis of their appearance into cancer and normal nuclei and used the location of detected nuclei to find cancer glands in prostate cancer.”
BackgroundLocal isotropic phase symmetry measure for detection of beta cells and lymphocytes
Kuse, M., Wang, Y.-F., Kalasannavar, V., Khan, M., & Rajpoot, N. (2012) · Journal of Pathology Informatics · doi:10.4103/2153-3539.92028 · Cited by 63 · cited 3×
“In our experiments (Section VII-C), the NEP (17) shows higher classification performance than that of the single-patch predictor (15).”
BackgroundSelf-citationBasavanhally, A. N., Feldman, M. D., Shih, N. N. C., Mies, C., Tomaszewski, J., Ganesan, S., & Madabhushi, A. (2012) · Journal of Pathology Informatics · doi:10.4103/2153-3539.92027 · Cited by 61 · cited 1×
“The spatial arrangement of these heterogeneous cell types has also been shown to be related to cancer grades [3], [4].”
BackgroundSharma, H., Zerbe, N., Heim, D., Wienert, S., Behrens, H., Hellwich, O., & Hufnagl, P. (2015) · International Conference on Computer Vision Theory and Applications · doi:10.5220/0005247900370046 · Cited by 61 · cited 1×
“[24] proposed nuclei segmentation and classification using AdaBoost classifier using intensity, morphological and texture features.”
MethodNuclear pleomorphism scoring by selective cell nuclei detection
Dalle, J.-R., Li, H., Huang, Z., Leow, W. K., Racoceanu, D., & Putti, T. C. (2009) · Cited by 53
A novel texture descriptor for detection of glandular structures in colon histology images
Sirinukunwattana, K., Snead, D. R. J., & Rajpoot, N. (2015) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2082010 · Cited by 41 · cited 3×
“The superpixel descriptor [37] was originally devised to distinguish area with different histologic pattern.”
MethodInfluentialSelf-citationClassification of potential nuclei in prostate histology images using shape manifold learning
Arif, M., & Rajpoot, N. (2007) · International Conference on Machine Vision · doi:10.1109/icmv.2007.4469283 · Cited by 35 · cited 1×
“Shape features have also been used in an unsupervised manifold learning framework [23] to discriminate between normal and malignant nuclei in prostate histology images.”
MethodSelf-citationMedical Image Computing and Computer Assisted Intervention – MICCAI 2024
Linguraru, M. G., Dou, Q., Feragen, A., Giannarou, S., Glocker, B., Lekadir, K., & Schnabel, J. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2 · Cited by 21
Research works printed but not linked: 1
- cruz-roa, 2013, Medical Image Computing and Computer-Assisted Intervention-MICCAI 2013, 403
Cited by
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
“An interesting example is the paper by Sirinukunwattana et al. (2016) , which details a method directly predicting the center locations of nuclei and shows that this outperforms classification- based center localization.”
MethodReview of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Deep learning in medical image analysis
Shen, D., Wu, G., & Suk, H.-I. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044442 · Cited by 4,968
“(100) proposed a spatially constrained CNN (SC-CNN) to detect and classify nuclei in histopathology images.”
BackgroundCoudray, N., Ocampo, P. S., Sakellaropoulos, T., Narula, N., Snuderl, M., Fenyö, D., Moreira, A. L., Razavian, N., & Tsirigos, A. (2018) · Nature Medicine · doi:10.1038/s41591-018-0177-5 · Cited by 2,930
A survey on deep learning and its applications
Dong, S., Wang, P., & Abbas, K. (2021) · Computer Science Review · doi:10.1016/j.cosrev.2021.100379 · Cited by 1,800
Metadata
- Authors
- Sirinukunwattana, K., Raza, S. E. A., Tsang, Y.-W., Snead, D. R. J., Cree, I. A., & Rajpoot, N. M.
- Affiliations
- University of Warwick (GB)
- University Hospital Coventry (GB)
- ORCID
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2016-05
- Volume
- 35
- Issue
- 5
- Pages
- 1196-1206
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2312404985
- Cited by
- 1,224
- References
- 41
- Citations by year
- 2026: 36 · 2025: 66 · 2024: 107 · 2023: 131 · 2022: 153 · 2021: 173 · 2020: 163 · 2019: 172 · 2018: 121 · 2017: 85 · 2016: 15
- FWCI
- 106.55
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- Qatar University QSpace (Qatar University) (submittedVersion)
- Funders
- Qatar National Research Fund (NPRP5-1345-1-228)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Imaging for Blood Diseases
- Keywords
- nucleus detection, colon cancer, colorectal adenocarcinoma, histopathological images, H&E staining, whole slide images, cell nucleus
- MeSH terms
- Machine Learning, Cell Nucleus, Colon, Colonic Neoplasms, Histocytochemistry, Humans, Image Interpretation, Computer-Assisted, Neural Networks, Computer, Cell Proliferation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.