The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
UNet++: Redesigning Skip Connections to Exploit Multiscale Features in Image Segmentation
Open accessAbstract
The state-of-the-art models for medical image segmentation are variants of U-Net and fully convolutional networks (FCN). Despite their success, these models have two limitations: (1) their optimal depth is apriori unknown, requiring extensive architecture search or inefficient ensemble of models of varying depths; and (2) their skip connections impose an unnecessarily restrictive fusion scheme, forcing aggregation only at the same-scale feature maps of the encoder and decoder sub-networks. To overcome these two limitations, we propose UNet++, a new neural architecture for semantic and instance segmentation, by (1) alleviating the unknown network depth with an efficient ensemble of U-Nets of varying depths, which partially share an encoder and co-learn simultaneously using deep supervision; (2) redesigning skip connections to aggregate features of varying semantic scales at the decoder sub-networks, leading to a highly flexible feature fusion scheme; and (3) devising a pruning scheme to accelerate the inference speed of UNet++. We have evaluated UNet++ using six different medical image segmentation datasets, covering multiple imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and electron microscopy (EM), and demonstrating that (1) UNet++ consistently outperforms the baseline models for the task of semantic segmentation across different datasets and backbone architectures; (2) UNet++ enhances segmentation quality of varying-size objects-an improvement over the fixed-depth U-Net; (3) Mask RCNN++ (Mask R-CNN with UNet++ design) outperforms the original Mask R-CNN for the task of instance segmentation; and (4) pruned UNet++ models achieve significant speedup while showing only modest performance degradation. Our implementation and pre-trained models are available at https://github.com/MrGiovanni/UNetPlusPlus.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
Milletari, F., Navab, N., & Ahmadi, S.-A. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.79 · Cited by 11,931 · cited 4×
“We choose U-Net (or V-Net for 3D) because it is a common performance baseline for image segmentation.”
BackgroundMethodInfluentialFully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665 · cited 1×
“Skip connections have further been used in modern neural architectures such as residual networks [8], [44] and dense networks [9], facilitating the gradient flow and improving the overall performance of classification networks.”
MethodUNet++: A Nested U-Net Architecture for Medical Image Segmentation
Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N., & Liang, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00889-5_1 · Cited by 9,610
Self-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781
Learning Transferable Architectures for Scalable Image Recognition
Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00907 · Cited by 6,249
Deep High-Resolution Representation Learning for Human Pose Estimation
Sun, K., Xiao, B., Liu, D., & Wang, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00584 · Cited by 5,984
Shin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528162 · Cited by 5,759
Deep learning in medical image analysis
Shen, D., Wu, G., & Suk, H.-I. (2017) · Annual Review of Biomedical Engineering · doi:10.1146/annurev-bioeng-071516-044442 · Cited by 4,968
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation
Lin, G., Milan, A., Shen, C., & Reid, I. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.549 · Cited by 3,287
Holistically-Nested Edge Detection
Xie, S., & Tu, Z. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.164 · Cited by 3,096
A Survey on Multi-Task Learning
Zhang, Y., & Yang, Q. (2017) · arXiv · doi:10.48550/arxiv.1707.08114 · Cited by 2,910
Armato, S. G., McLennan, G., Bidaut, L., McNitt‐Gray, M. F., Meyer, C. R., Reeves, A. P., Zhao, B., Aberle, D. R., Henschke, C. I., Hoffman, E. A., et al. (2011) · Medical Physics · doi:10.1118/1.3528204 · Cited by 2,887 · cited 2×
“The dataset is provided by the Lung Image Database Consortium image collection (LIDC-IDRI) [33] and consists of 1018 cases collected by seven academic centers and eight medical imaging companies.”
BackgroundA Survey on Multi-Task Learning
Zhang, Y., & Yang, Q. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3070203 · Cited by 2,878
Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations
Sudre, C. H., Li, W., Vercauteren, T., Ourselin, S., & Jorge Cardoso, M. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-67558-9_28 · Cited by 2,829
U-Net: deep learning for cell counting, detection, and morphometry
Falk, T., Mai, D., Bensch, R., Çiçek, Ö., Abdulkadir, A., Marrakchi, Y., Böhm, A., Deubner, J., Jäckel, Z., Seiwald, K., et al. (2019) · Nature Methods · doi:10.1038/s41592-018-0261-2 · Cited by 1,902
Progressive Neural Architecture Search
Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_2 · Cited by 1,833
ICNet for Real-Time Semantic Segmentation on High-Resolution Images
Zhao, H., Qi, X., Shen, X., Shi, J., & Jia, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_25 · Cited by 1,709
Yu, F., Wang, D., Shelhamer, E., & Darrell, T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00255 · Cited by 1,516
Hypercolumns for object segmentation and fine-grained localization
Hariharan, B., Arbelaez, P., Girshick, R., & Malik, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298642 · Cited by 1,511
LinkNet: Exploiting encoder representations for efficient semantic segmentation
Chaurasia, A., & Culurciello, E. (2017) · 2017 IEEE Visual Communications and Image Processing (VCIP) · doi:10.1109/vcip.2017.8305148 · Cited by 1,507
Deep Learning: A Primer for Radiologists
Chartrand, G., Cheng, P. M., Vorontsov, E., Drozdzal, M., Turcotte, S., Pal, C. J., Kadoury, S., & Tang, A. (2017) · RadioGraphics · doi:10.1148/rg.2017170077 · Cited by 1,115
Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation
Tajbakhsh, N., Jeyaseelan, L., Li, Q., Chiang, J. N., Wu, Z., & Ding, X. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101693 · Cited by 1,095 · cited 1×
“THE encoder-decoder networks are widely used in modern semantic and instance segmentation models [1]–[6].”
MethodSelf-citationAuto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
Liu, C., Chen, L.-C., Schroff, F., Adam, H., Hua, W., Yuille, A. L., & Fei-Fei, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00017 · Cited by 1,025
Lee, C., Xie, S., Gallagher, P., Zhang, Z., & Zhuowen Tu, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.5185 · Cited by 993
The Importance of Skip Connections in Biomedical Image Segmentation
Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury, S., & Pal, C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46976-8_19 · Cited by 984
End-to-End Change Detection for High Resolution Satellite Images Using Improved UNet++
Peng, D., Zhang, Y., & Guan, H. (2019) · Remote Sensing · doi:10.3390/rs11111382 · Cited by 804
High-Resolution Representations for Labeling Pixels and Regions
Sun, K., Zhao, Y. S., Jiang, B., Cheng, T., Xiao, B., Liu, D., Mu, Y., Wang, X., Liu, W., & Wang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04514 · Cited by 614
3D deeply supervised network for automated segmentation of volumetric medical images
Dou, Q., Yu, L., Chen, H., Jin, Y., Yang, X., Qin, J., & Heng, P.-A. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.05.001 · Cited by 613
Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes
Pohlen, T., Hermans, A., Mathias, M., & Leibe, B. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.353 · Cited by 584
Cardona, A., Saalfeld, S., Preibisch, S., Schmid, B., Cheng, A., Pulokas, J., Tomancak, P., & Hartenstein, V. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000502 · Cited by 372
Selective Feature Aggregation Network with Area-Boundary Constraints for Polyp Segmentation
Fang, Y., Chen, C., Yuan, Y., & Tong, K.-Y. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32239-7_34 · Cited by 353
3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes
Dou, Q., Chen, H., Jin, Y., Yu, L., Qin, J., & Heng, P.-A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46723-8_18 · Cited by 344
The Virtual Skeleton Database: An Open Access Repository for Biomedical Research and Collaboration
Kistler, M., Bonaretti, S., Pfahrer, M., Niklaus, R., & Büchler, P. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2930 · Cited by 303
Ciompi, F., de Hoop, B., van Riel, S. J., Chung, K., Scholten, E. T., Oudkerk, M., de Jong, P. A., Prokop, M., & Ginneken, B. V. (2015) · Medical Image Analysis · doi:10.1016/j.media.2015.08.001 · Cited by 289
Learning to Segment Every Thing
Hu, R., Dollar, P., He, K., Darrell, T., & Girshick, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00445 · Cited by 281
Jiang, J., Hu, Y.-C., Liu, C.-J., Halpenny, D., Hellmann, M. D., Deasy, J. O., Mageras, G., & Veeraraghavan, H. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2857800 · Cited by 268
Residual Conv-Deconv Grid Network for Semantic Segmentation
Fourure, D., Emonet, R., Fromont, E., Muselet, D., Tremeau, A., & Wolf, C. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.181 · Cited by 223
Deeply-supervised CNN for prostate segmentation
Zhu, Q., Du, B., Turkbey, B., Choyke, P. L., & Yan, P. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7965852 · Cited by 191
MDU-Net: multi-scale densely connected U-Net for biomedical image segmentation
Zhang, J., Zhang, Y., Jin, Y., Xu, J., & Xu, X. (2023) · Health Information Science and Systems · doi:10.1007/s13755-022-00204-9 · Cited by 183
Partial Order Pruning: For Best Speed/Accuracy Trade-Off in Neural Architecture Search
Li, X., Zhou, Y., Pan, Z., & Feng, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00936 · Cited by 177
Customizable Architecture Search for Semantic Segmentation
Zhang, Y., Qiu, Z., Liu, J., Yao, T., Liu, D., & Mei, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01191 · Cited by 174
Progressive Neural Architecture Search
Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. (2017) · arXiv · doi:10.48550/arxiv.1712.00559 · Cited by 171 · cited 1×
“hindered by the limited computational resources [19]–[23].”
Collaborative Learning for Deep Neural NetworksPreprint
Song, G., & Chai, W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.11761 · Cited by 167
Deep Contextual Networks for Neuronal Structure Segmentation
Chen, H., Qi, X., Cheng, J., & Heng, P. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10141 · Cited by 160
Learning Contextual and Attentive Information for Brain Tumor Segmentation
Zhou, C., Chen, S., Ding, C., & Tao, D. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11726-9_44 · Cited by 136
Deep Learning in Medical Image Analysis
Zhang, Y., Górriz, J. M., & Dong, Z. (2021) · MDPI eBooks · doi:10.3390/books978-3-0365-1470-3 · Cited by 117
Pulmonary Vessel Segmentation Based on Orthogonal Fused U-Net++ of Chest CT Images
Cui, H., Liu, X., & Huang, N. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32226-7_33 · Cited by 74
Multiscale dense convolutional neural network for DSA cerebrovascular segmentation
Cai, M., Sun, K., Guan, S., et al. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.10.035 · Cited by 71
Song, T., Meng, F., Rodríguez‐Patón, A., Li, P., Zheng, P., & Wang, X. (2019) · IEEE Access · doi:10.1109/access.2019.2953934 · Cited by 56
Zyuzin, V., & Chumarnaya, T. V. (2019) · Ural Symposium on Biomedical Engineering, Radioelectronics and Information Technology (USBEREIT) · doi:10.1109/usbereit.2019.8736616 · Cited by 47
Yang, C., & Gao, F. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32239-7_28 · Cited by 43
Zhang, Y., Gong, W., Sun, J., & Li, W. (2019) · Remote Sensing · doi:10.3390/rs11161897 · Cited by 42
Automatical Segmentation of Pelvic Organs After Hysterectomy by using Dilated Convolution U-Net++
Wu, S., Wang, Z., Liu, C., Zhu, C., Wu, S., & Xiao, K. (2019) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c.2019.00074 · Cited by 27
ErrorNet: Learning Error Representations from Limited Data to Improve Vascular Segmentation
Tajbakhsh, N., Lai, B., Ananth, S. P., et al. (2020) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi45749.2020.9098451 · Cited by 22
Self-citationAn Improved Framework Called Du++ Applied to Brain Tumor Segmentation
Chen, F., Ding, Y., Wu, Z., Wu, D., & Wen, J.-M. (2018) · International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP) · doi:10.1109/iccwamtip.2018.8632559 · Cited by 8
Fang, J., Zhang, Y., Xie, K., et al. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32956-3_16 · Cited by 6
Cited by
Chen, J., Mei, J., Li, X., Lu, Y., Yu, Q., Wei, Q., Luo, X., Xie, Y., Adeli, E., Wang, Y., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103280 · Cited by 1,365
TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation
Zhang, Y., Liu, H., & Hu, Q. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-87193-2_2 · Cited by 1,312
Transformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,296
SNUNet-CD: A Densely Connected Siamese Network for Change Detection of VHR Images
Fang, S., Li, K., Shao, J., & Li, Z. (2022) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2021.3056416 · Cited by 1,182
Recent advances and clinical applications of deep learning in medical image analysis
Chen, X., Wang, X., Zhang, K., Fung, K.-M., Thai, T. C., Moore, K., Mannel, R. S., Liu, H., Zheng, B., & Qiu, Y. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102444 · Cited by 1,080
Metadata
- Authors
- Zhou, Z., Siddiquee, M. M. R., Tajbakhsh, N., & Liang, J.
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2020-06
- Volume
- 39
- Issue
- 6
- Pages
- 1856-1867
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2996290406
- Cited by
- 4,402
- References
- 70
- FWCI
- 106.81
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.