Automated Melanoma Recognition in Dermoscopy Images via Very Deep Residual Networks
Abstract
Automated melanoma recognition in dermoscopy images is a very challenging task due to the low contrast of skin lesions, the huge intraclass variation of melanomas, the high degree of visual similarity between melanoma and non-melanoma lesions, and the existence of many artifacts in the image. In order to meet these challenges, we propose a novel method for melanoma recognition by leveraging very deep convolutional neural networks (CNNs). Compared with existing methods employing either low-level hand-crafted features or CNNs with shallower architectures, our substantially deeper networks (more than 50 layers) can acquire richer and more discriminative features for more accurate recognition. To take full advantage of very deep networks, we propose a set of schemes to ensure effective training and learning under limited training data. First, we apply the residual learning to cope with the degradation and overfitting problems when a network goes deeper. This technique can ensure that our networks benefit from the performance gains achieved by increasing network depth. Then, we construct a fully convolutional residual network (FCRN) for accurate skin lesion segmentation, and further enhance its capability by incorporating a multi-scale contextual information integration scheme. Finally, we seamlessly integrate the proposed FCRN (for segmentation) and other very deep residual networks (for classification) to form a two-stage framework. This framework enables the classification network to extract more representative and specific features based on segmented results instead of the whole dermoscopy images, further alleviating the insufficiency of training data. The proposed framework is extensively evaluated on ISBI 2016 Skin Lesion Analysis Towards Melanoma Detection Challenge dataset. Experimental results demonstrate the significant performance gains of the proposed framework, ranking the first in classification and the second in segmentation among 25 teams and 28 teams, respectively. This study corroborates that very deep CNNs with effective training mechanisms can be employed to solve complicated medical image analysis tasks, even with limited training data.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Siegel, R. L., Miller, K. D., & Jemal, A. (2016) · CA: A Cancer Journal for Clinicians · doi:10.3322/caac.21332 · Cited by 16,003 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
(2010) · Cited by 14,134 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741 · see citing works
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542 · see citing works
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244 · see citing works
Shin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528162 · Cited by 5,759 · see citing works
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · see citing works
Holistically-Nested Edge Detection
Xie, S., & Tu, Z. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.164 · Cited by 3,401 · see citing works
Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
Tajbakhsh, N., Shin, J. Y., Gurudu, S. R., Hurst, R. T., Kendall, C. B., Gotway, M. B., & Liang, J. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2535302 · Cited by 3,277 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714 · see citing works
Final Version of the American Joint Committee on Cancer Staging System for Cutaneous Melanoma
Balch, C. M., Buzaid, A. C., Soong, S., Atkins, M. B., Cascinelli, N., Coit, D. G., Fleming, I. D., Gershenwald, J. E., Houghton, A. N., Kirkwood, J. M., et al. (2001) · Journal of Clinical Oncology · doi:10.1200/jco.2001.19.16.3635 · Cited by 2,627 · see citing works
Codella, N. C. F., Gutman, D., Celebi, M. E., Helba, B., Marchetti, M. A., Dusza, S. W., Kalloo, A., Liopyris, K., Mishra, N., Kittler, H., et al. (2018) · 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) · doi:10.1109/isbi.2018.8363547 · Cited by 2,620 · see citing works
Conditional Random Fields as Recurrent Neural Networks
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., & Torr, P. H. S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.179 · Cited by 2,386 · cited 1× · see citing works
“In the future, we shall investigate to integrate Bayesian learning, especially probabilistic graphical models [48], [49]”
MethodWide Residual NetworksPreprint
Zagoruyko, S., & Komodakis, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07146 · Cited by 1,850 · see citing works
Greenspan, H. K., van Ginneken, B., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2553401 · Cited by 1,704 · see citing works
Deeply-Supervised NetsPreprint
Lee, C., Xie, S., Gallagher, P., Zhang, Z., & Tu, Z. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.5185 · Cited by 1,607 · see citing works
Lung pattern classification for interstitial lung diseases using a deep convolutional neural network
Anthimopoulos, M., Christodoulidis, S., Ebner, L., Christe, A., & Mougiakakou, S. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2535865 · Cited by 1,343 · see citing works
Instance-aware Semantic Segmentation via Multi-task Network Cascades
Dai, J., He, K., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.04412 · Cited by 1,273 · see citing works
Setio, A. A. A., Ciompi, F., Litjens, G., Gerke, P., Jacobs, C., van Riel, S. J., Wille, M. M. W., Naqibullah, M., Sanchez, C. I., & van Ginneken, B. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2536809 · Cited by 1,258 · see citing works
Instance-Aware Semantic Segmentation via Multi-task Network Cascades
Dai, J., He, K., & Sun, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.343 · Cited by 1,251 · see citing works
An empirical evaluation of deep architectures on problems with many factors of variation
Larochelle, H., Erhan, D., Courville, A. C., Bergstra, J., & Bengio, Y. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273556 · Cited by 1,024 · see citing works
Gland segmentation in colon histology images: The glas challenge contest
Sirinukunwattana, K., Pluim, J. P. W., Chen, H., Qi, X., Heng, P., Guo, Y. B., Wang, L. Y., Matuszewski, B. J., Bruni, E., Sanchez, U., et al. (2016) · Medical Image Analysis · doi:10.1016/j.media.2016.08.008 · Cited by 989 · see citing works
Self-citationDeep Learning using Linear Support Vector MachinesPreprint
Tang, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1306.0239 · Cited by 891 · see citing works
A methodological approach to the classification of dermoscopy images
Celebi, M. E., Kingravi, H. A., Uddin, B., Iyatomi, H., Aslandogan, Y. A., Stoecker, W. V., & Moss, R. H. (2007) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2007.01.003 · Cited by 697 · see citing works
Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks
Dou, Q., Chen, H., Yu, L., Zhao, L., Qin, J., Wang, D., Mok, V. C., Shi, L., & Heng, P.-A. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528129 · Cited by 679 · see citing works
Self-citationDCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation
Chen, H., Qi, X., Yu, L., & Heng, P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.273 · Cited by 671 · see citing works
Self-citationAutomated melanoma recognition
Ganster, H., Pinz, P., Röhrer, R., Wildling, E., Binder, M., & Kittler, H. (2001) · IEEE Transactions on Medical Imaging · doi:10.1109/42.918473 · Cited by 633 · see citing works
Computational limitations of small-depth circuits
Håstad, J. (1987) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 452 · see citing works
Comparison of Segmentation Methods for Melanoma Diagnosis in Dermoscopy Images
Silveira, M., Nascimento, J. C., Marques, J. S., Marcal, A. R. S., Mendonça, T., Yamauchi, S., Maeda, J., & Rozeira, J. (2009) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2008.2011119 · Cited by 409 · see citing works
Deep Learning, Sparse Coding, and SVM for Melanoma Recognition in Dermoscopy Images
Codella, N., Cai, J., Abedini, M., Garnavi, R., Halpern, A. C., & Smith, J. R. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24888-2_15 · Cited by 400 · see citing works
Gutman, D. A., Codella, N., Celebi, E., Helba, B., Marchetti, M. A., Mishra, N. K., & Halpern, A. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.01397 · Cited by 365 · see citing works
Deep features to classify skin lesions
Kawahara, J., BenTaieb, A., & Hamarneh, G. (2016) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2016.7493528 · Cited by 356 · see citing works
Binder, M. (1995) · Archives of Dermatology · doi:10.1001/archderm.1995.01690150050011 · Cited by 355 · see citing works
Early detection and treatment of skin cancer.
Jerant, A. F., Johnson, J. A., Sheridan, C. D., et al. (2000) · PubMed · Cited by 316 · see citing works
Ballerini, L., Fisher, R. B., Aldridge, B., & Rees, J. (2012) · Lecture notes in computational vision and biomechanics · doi:10.1007/978-94-007-5389-1_4 · Cited by 289 · see citing works
Early detection and treatment of skin cancer
Trovitch, P. B., Gupte, A., & Çiftçi, K. (2002) · Cited by 267 · see citing works
On the power of small-depth threshold circuits
H�stad, J., & Goldmann, M. (1991) · Computational Complexity · doi:10.1007/bf01272517 · Cited by 253 · see citing works
Screening for malignant melanoma: A cost-effectiveness analysis
Freedberg, K. A., Geller, A. C., Miller, D. R., et al. (1999) · Journal of the American Academy of Dermatology · doi:10.1016/s0190-9622(99)70010-1 · Cited by 176 · see citing works
An ensemble classification approach for melanoma diagnosis
Schaefer, G., Krawczyk, B., Celebi, M. E., & Iyatomi, H. (2014) · Memetic Computing · doi:10.1007/s12293-014-0144-8 · Cited by 113 · see citing works
An Overview of Melanoma Detection in Dermoscopy Images Using Image Processing and Machine LearningPreprint
Mishra, N. K., & Celebi, M. E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.07843 · Cited by 105 · see citing works
Skin lesion classification using relative color features
Cheng, Y., Swamisai, R., Umbaugh, S. E., Moss, R. H., Stoecker, W. V., Teegala, S., & Srinivasan, S. K. (2007) · Skin Research and Technology · doi:10.1111/j.1600-0846.2007.00261.x · Cited by 104 · see citing works
Stanley, R. J., Stoecker, W. V., & Moss, R. H. (2007) · Skin Research and Technology · doi:10.1111/j.1600-0846.2007.00192.x · Cited by 96 · see citing works
Melanoma Recognition Using Representative and Discriminative Kernel Classifiers
Tommasi, T., Torre, E. L., & Caputo, B. (2006) · Lecture notes in computer science · doi:10.1007/11889762_1 · Cited by 40 · see citing works
Towards Bayesian Deep Learning: A SurveyPreprint
Wang, H., & Yeung, D. (2016) · arXiv (Cornell University) · Cited by 28 · see citing works
Gland Segmentation in Colon Histology Images: The GlaS Challenge ContestPreprint
Sirinukunwattana, K., Pluim, J. P. W., Chen, H., Qi, X., Heng, P., Guo, Y. B., Wang, L. Y., Matuszewski, B. J., Bruni, E., Sanchez, U., et al. (2016) · Zurich Open Repository and Archive (University of Zurich) · doi:10.3929/ethz-b-000127185 · Cited by 10 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332 · see citing works
Opportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310 · see citing works
AI in Medical Imaging Informatics: Current Challenges and Future Directions
Panayides, A. S., Amini, A., Filipovic, N. D., Sharma, A., Tsaftaris, S. A., Young, A., Foran, D., Do, N., Golemati, S., Kurc, T., et al. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2991043 · Cited by 713 · see citing works
Automatic skin lesion segmentation using deep fully convolutional networks with jaccard distance.
Yuan, Y., Chao, M., & Lo, Y.-C. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2695227 · Cited by 687 · see citing works
Skin cancer detection: A review using deep learning techniques
Dildar, M., Akram, S., Irfan, M., Khan, H. U., Ramzan, M., Mahmood, A. R., Alsaiari, S. A., Saeed, A. H. M., Alraddadi, M. O., & Mahnashi, M. H. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18105479 · Cited by 597 · see citing works
Seven-point checklist and skin lesion classification using multitask multimodal neural nets
Kawahara, J., Daneshvar, S., Argenziano, G., & Hamarneh, G. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2018.2824327 · Cited by 584 · see citing works
SV-RCNet: Workflow Recognition From Surgical Videos Using Recurrent Convolutional Network
Jin, Y., Dou, Q., Chen, H., Yu, L., Qin, J., Fu, C.-W., & Heng, P.-A. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2787657 · Cited by 316 · see citing works
Self-citationNaeem, A., Farooq, M. S., Khelifi, A., & Abid, A. (2020) · IEEE Access · doi:10.1109/access.2020.3001507 · Cited by 189 · see citing works
Skin disease diagnosis with deep learning: a review
Li, H., Pan, Y., Zhao, J., & Zhang, L. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.08.096 · Cited by 178 · see citing works
A Convolutional Neural Network Framework for Accurate Skin Cancer Detection
Thurnhofer-Hemsi, K., & Domínguez, E. (2021) · Neural Processing Letters · doi:10.1007/s11063-020-10364-y · Cited by 157 · see citing works
FusionM4Net: a multi-stage multi-modal learning algorithm for multi-label skin lesion classification
Tang, P., Yan, X., Nan, Y., Xiang, S., Krammer, S., & Lasser, T. (2022) · Medical Image Analysis · doi:10.1016/j.media.2021.102307 · Cited by 131 · see citing works
Melanoma Lesion Detection and Segmentation Using YOLOv4-DarkNet and Active Contour
Albahli, S., Nida, N., Irtaza, A., Yousaf, M. H., & Mahmood, M. T. (2020) · IEEE Access · doi:10.1109/access.2020.3035345 · Cited by 126 · see citing works
The impact of AI on modern oncology from early detection to personalized cancer treatment
Li, J., Zhang, L., Yu, Z., Bao, Z., Li, D., & Wang, L. (2026) · npj Precision Oncology · doi:10.1038/s41698-026-01276-6 · Cited by 24 · see citing works
Liu, C., Qiao, M., Jiang, F., Guo, Y., Jin, Z., & Wang, Y. (2021) · Medical Physics · doi:10.1002/mp.15172 · Cited by 22 · see citing works
Multiscale ensemble of convolutional neural networks for skin lesion classification
Liu, Y., Wang, Z., Li, Z., Li, J., Li, T., Chen, P., & Liang, R. (2021) · IET Image Processing · doi:10.1049/ipr2.12214 · Cited by 18 · see citing works
- Authors
- Yu, L., Chen, H., Dou, Q., Qin, J., & Heng, P.-A.
- Affiliations
- Chinese University of Hong Kong (HK)
- Hong Kong Polytechnic University (HK)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2017-04
- Volume
- 36
- Issue
- 4
- Pages
- 994-1004
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2564782580
- Cited by
- 1,140
- References
- 56
- Citations by year
- 2026: 47 · 2025: 89 · 2024: 131 · 2023: 141 · 2022: 167 · 2021: 191 · 2020: 133 · 2019: 133 · 2018: 82 · 2017: 25 · 2015: 1
- FWCI
- 42.91
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Funders
- Research Grants Council of the Hong Kong Special Administrative Region (CUHK 14203115, CUHK 14202514)
- Guangdong Natural Science Foundation (2016A030313047)
- Topics
- Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- melanoma detection, dermoscopic images, skin lesion segmentation, multi-scale contextual information, limited training data, residual learning, overfitting, skin lesion classification
- MeSH terms
- Humans, Melanoma, Artifacts, Neural Networks, Computer, Dermoscopy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
