UNETR++: Delving Into Efficient and Accurate 3D Medical Image Segmentation
Open accessThe DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Abstract
Owing to the success of transformer models, recent works study their applicability in 3D medical segmentation tasks. Within the transformer models, the self-attention mechanism is one of the main building blocks that strives to capture long-range dependencies, compared to the local convolutional-based design. However, the self-attention operation has quadratic complexity which proves to be a computational bottleneck, especially in volumetric medical imaging, where the inputs are 3D with numerous slices. In this paper, we propose a 3D medical image segmentation approach, named UNETR++, that offers both high-quality segmentation masks as well as efficiency in terms of parameters, compute cost, and inference speed. The core of our design is the introduction of a novel efficient paired attention (EPA) block that efficiently learns spatial and channel-wise discriminative features using a pair of inter-dependent branches based on spatial and channel attention. Our spatial attention formulation is efficient and has linear complexity with respect to the input. To enable communication between spatial and channel-focused branches, we share the weights of query and key mapping functions that provide a complimentary benefit (paired attention), while also reducing the complexity. Our extensive evaluations on five benchmarks, Synapse, BTCV, ACDC, BraTS, and Decathlon-Lung, reveal the effectiveness of our contributions in terms of both efficiency and accuracy. On Synapse, our UNETR++ sets a new state-of-the-art with a Dice Score of 87.2%, while significantly reducing parameters and FLOPs by over 71%, compared to the best method in the literature. Our code and models are available at: https://tinyurl.com/2p87x5xn.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · arXiv · doi:10.48550/arxiv.1802.02611 · Cited by 20,063 · cited 1×
“Further, several efforts in the literature have been made to encode holistic contextual information within CNN-based frameworks using, e.g ., image pyramids [35], large kernels [26], dilated convolution [6], and deformable convolution [20].”
BackgroundEncoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_49 · Cited by 20,054
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438
V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
Milletari, F., Navab, N., & Ahmadi, S.-A. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.79 · Cited by 11,931 · cited 3×
“Our loss function is based on a summation of the commonly used soft dice loss [25] and cross-entropy loss to simultaneously leverage the benefits of both complementary loss functions.”
BackgroundMethodnnU-Net: a self-configuring method for deep learning-based biomedical image segmentation
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021) · Nature Methods · doi:10.1038/s41592-020-01008-z · Cited by 10,477
UNet++: A Nested U-Net Architecture for Medical Image Segmentation
Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N., & Liang, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00889-5_1 · Cited by 9,610
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., & Xie, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01167 · Cited by 8,494
3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation
Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T., & Ronneberger, O. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46723-8_49 · Cited by 7,945 · cited 2×
“In the case of 3D medical image segmentation [8,10,11,25, 31], the full volumetric image is typically processed as a sequence of 2D slices.”
BackgroundMethodThe multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781
CBAM: Convolutional Block Attention Module
Woo, S., Park, J., Lee, J.-Y., & Kweon, I. S. (2018) · arXiv · doi:10.48550/arxiv.1807.06521 · Cited by 6,261 · cited 1×
“As illustrated, UNETR++ has a better delineation of the boundaries of different organs without under/over-segmenting, thus suggesting that UNETR++ encodes enriched inter-dependent spatial and channel features within the proposed EPA block.”
BackgroundSwin-Unet: Unet-like pure transformer for medical image segmentation
Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., & Wang, M. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-25066-8_9 · Cited by 5,640
Attention U-Net: Learning Where to Look for the Pancreas
Oktay, O., Schlemper, J., Folgoc, L. L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N. Y., Kainz, B., et al. (2018) · arXiv · doi:10.48550/arxiv.1804.03999 · Cited by 5,458
TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021) · arXiv · doi:10.48550/arxiv.2102.04306 · Cited by 4,206 · cited 2×
“TransUNet [5] combines transformers and the U-Net architecture, where transformers encode the embedded image patches from convolution features and the decoder combines the upsampled encoded features with high-resolution CNN features for localization.”
BackgroundUNETR: Transformers for 3D Medical Image Segmentation
Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H. R., & Xu, D. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv51458.2022.00181 · Cited by 3,586 · cited 10×
“Hatamizadeh et al. [13] introduce a 3D hybrid model, UNETR, that combines the long-range spatial dependencies of transformers with the CNN’s inductive bias into a “U-shaped” encoder-decoder architecture.”
BackgroundMethodResultInfluentialUNETR: Transformers for 3D Medical Image Segmentation
Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H., & Xu, D. (2021) · arXiv · doi:10.48550/arxiv.2103.10504 · Cited by 3,580
UNet 3+: A Full-Scale Connected UNet for Medical Image Segmentation
Huang, H., Lin, L., Tong, R., Hu, H., Zhang, Q., Iwamoto, Y., Han, X., Chen, Y.-W., & Wu, J. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053405 · Cited by 2,774
Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., Yang, X., Heng, P.-A., Cetin, I., Lekadir, K., Camara, O., Gonzalez Ballester, M. A., et al. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2837502 · Cited by 2,431 · cited 2×
“…on five datasets: Synapse for Multi-organ CT Segmentation [19], BTCV for Multi-organ CT Segmentation [19], ACDC for Automated Cardiac Diagnosis [1], Brain Tumor Segmentation (BraTS) [24] and the Medical Segmentation Decathlon-Lung [30]. of right ventricle (RV), left ventricle (LV) and…”
BackgroundMethodSwin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images
Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H. R., & Xu, D. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-08999-2_22 · Cited by 2,273
Large Kernel Matters — Improve Semantic Segmentation by Global Convolutional Network
Peng, C., Zhang, X., Yu, G., Luo, G., & Sun, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.189 · Cited by 1,750 · cited 1×
“Further, several efforts in the literature have been made to encode holistic contextual information within CNN-based frameworks using, e.g ., image pyramids [35], large kernels [26], dilated convolution [6], and deformable convolution [20].”
BackgroundReformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435 · cited 1×
“Few recent works have explored alleviating the complexity issue of standard self-attention operation within transformer frameworks [7, 18, 23, 30].”
BackgroundTransFuse: Fusing Transformers and CNNs for Medical Image Segmentation
Zhang, Y., Liu, H., & Hu, Q. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-87193-2_2 · Cited by 1,312
Medical Transformer: Gated Axial-Attention for Medical Image Segmentation
Valanarasu, J. M. J., Oza, P., Hacihaliloglu, I., & Patel, V. M. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-87193-2_4 · Cited by 1,248
Generating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063
TransBTS: Multimodal Brain Tumor Segmentation Using Transformer
Wang, W., Chen, C., Ding, M., Yu, H., Zha, S., & Li, J. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-87193-2_11 · Cited by 1,045
DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation
Lin, A., Chen, B., Xu, J., Zhang, Z., Lu, G., & Zhang, D. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3178991 · Cited by 882
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867 · cited 3×
“To investigate the scalability of UNETR++, we designed an experiment with feature maps of size [64 , 128 , 256 , 512] instead of [32 , 64 , 128 , 256] on the BTCV dataset.”
BackgroundMethodnnFormer: Volumetric Medical Image Segmentation via a 3D Transformer
Zhou, H.-Y., Guo, J., Zhang, Y., Han, X., Yu, L., Wang, L., & Yu, Y. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3293771 · Cited by 735
Automatic Multi-Organ Segmentation on Abdominal CT With Dense V-Networks
Gibson, E., Giganti, F., Hu, Y., Bonmati, E., Bandula, S., Gurusamy, K., Davidson, B., Pereira, S. P., Clarkson, M. J., & Barratt, D. C. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2806309 · Cited by 697
CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation
Xie, Y., Zhang, J., Shen, C., & Xia, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.03024 · Cited by 654
CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation
Xie, Y., Zhang, J., Shen, C., & Xia, Y. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-87199-4_16 · Cited by 653
Simpson, A. L., Antonelli, M., Bakas, S., Bilello, M., Farahani, K., van Ginneken, B., Kopp‐Schneider, A., Landman, B. A., Litjens, G. J. S., Menze, B. H., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.09063 · Cited by 643
MISSFormer: An Effective Transformer for 2D Medical Image Segmentation
Huang, X., Deng, Z., Li, D., Yuan, X., & Fu, Y. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3230943 · Cited by 594
Metrics reloaded: recommendations for image analysis validation
Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M. D., Buettner, F., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., et al. (2024) · Nature Methods · doi:10.1038/s41592-023-02151-z · Cited by 470
MONAI: An open-source framework for deep learning in healthcare
Cardoso, M. J., Li, W., Brown, R., Ma, N., Kerfoot, E., Wang, Y., Murrey, B., Myronenko, A., Zhao, C. H., Yang, D., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.02701 · Cited by 432
Edgenext: Efficiently amalgamated CNN-transformer architecture for mobile vision applications
Maaz, M., Shaker, A., Cholakkal, H., Khan, S., Zamir, S. W., Anwer, R. M., & Shahbaz Khan, F. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-25082-8_1 · Cited by 407
Self-citationMedNeXt: Transformer-Driven Scaling of ConvNets for Medical Image Segmentation
Roy, S., Koehler, G., Ulrich, C., Baumgartner, M., Petersen, J., Isensee, F., Jäger, P. F., & Maier-Hein, K. H. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-43901-8_39 · Cited by 347
3D Deeply Supervised Network for Automatic Liver Segmentation from CT Volumes
Dou, Q., Chen, H., Jin, Y., Yu, L., Qin, J., & Heng, P.-A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46723-8_18 · Cited by 344
Cai, S., Tian, Y., Lui, H., Zeng, H., Wu, Y., & Chen, G. (2020) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-19-1090 · Cited by 293
Segmentation Outside the Cranial Vault Challenge
harrigr (2015) · Synapse · doi:10.7303/syn3193805 · Cited by 259
A CBAM Based Multiscale Transformer Fusion Approach for Remote Sensing Image Change Detection
Wang, W., Tan, X., Zhang, P., & Wang, X. (2022) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2022.3198517 · Cited by 208
Deeply-supervised CNN for prostate segmentation
Zhu, Q., Du, B., Turkbey, B., Choyke, P. L., & Yan, P. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7965852 · Cited by 191
ReSTR: Convolution-free Referring Image Segmentation Using Transformers
Kim, N., Kim, D., Kwak, S., Lan, C., & Zeng, W. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01761 · Cited by 167
Common Limitations of Image Processing Metrics: A Picture Story
Reinke, A., Tizabi, M. D., Sudre, C. H., Eisenmann, M., Rädsch, T., Baumgartner, M., Acion, L., Antonelli, M., Arbel, T., Bakas, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2104.05642 · Cited by 126 · cited 1×
“In this section, we provide additional details regarding: • Implementation Details (Appendix A) • Qualitative Results (Appendix B) • Ablations (Appendix C) • Discussion (Appendix D)”
BackgroundHierarchical 3D fully convolutional networks for multi-organ segmentation
Roth, H. R., Oda, H., Hayashi, Y., Oda, M., Shimizu, N., Fujiwara, M., Misawa, K., & Mori, K. (2017) · arXiv · doi:10.48550/arxiv.1704.06382 · Cited by 111 · cited 1×
“Roth et al. [29] propose a multi-scale 3D fully convolution network to learn representations from varying resolutions for multi-organ segmentation.”
BackgroundPGD-UNet: A Position-Guided Deformable Network for Simultaneous Segmentation of Organs and Tumors
Li, Z., Pan, H., Zhu, Y., & Qin, A. K. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206944 · Cited by 34
Cited by
Beyond Self-Attention: Deformable Large Kernel Attention for Medical Image Segmentation
Azad, R., Niggemeier, L., Hüttemann, M., Kazerouni, A., Aghdam, E. K., Velichko, Y., Bagci, U., & Merhof, D. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00132 · Cited by 254
3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation
Gong, S., Zhong, Y., Ma, W., Li, J., Wang, Z., Zhang, J., Heng, P.-A., & Dou, Q. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103324 · Cited by 156
“However, precise tumor segmentation is still challenging even for state-of-the-art segmentation networks such as nnU-Net [22], UNETER++ [42] and 3D UX-Net [43], because tumors usually have notable properties of small size, irregular shape, low contrast, and ambiguous boundaries.”
BackgroundMethodscAMAC: Self-supervised clustering of scRNA-seq data based on adaptive multi-scale autoencoder
Tan, D., Yang, C., Wang, J., Su, Y., & Zheng, C. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae068 · Cited by 19
OPUS-Rota5: A highly accurate protein side-chain modeling method with 3D-Unet and RotaFormer
Xu, G., Luo, Z., Yan, Y., Wang, Q., & Ma, J. (2024) · Structure · doi:10.1016/j.str.2024.03.015 · Cited by 8
Metadata
- Authors
- Shaker, A., Maaz, M., Rasheed, H., Khan, S., Yang, M.-H., & Shahbaz Khan, F.
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2024-09
- Volume
- 43
- Issue
- 9
- Pages
- 3377-3390
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W4396783545
- Cited by
- 500
- References
- 50
- FWCI
- 68.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.