CoChat
Sign Up
PreprintCited by 26Medical Image Segmentation TechniquesAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

SAM-Med2D

Open access
Cheng, Junlong, Ye, Jin, Deng, Zhongying, Chen, Jianpin, Li, Tianbin, Wang, Haoyu, Su, Yanzhou, Huang, Ziyan +7 more·2023·arXiv
DOI 10.48550/arxiv.2308.16184arXiv 2308.16184
Open this paper in CoChat — free accountOpen PDF

Abstract

The Segment Anything Model (SAM) represents a state-of-the-art research advancement in natural image segmentation, achieving impressive results with input prompts such as points and bounding boxes. However, our evaluation and recent research indicate that directly applying the pretrained SAM to medical image segmentation does not yield satisfactory performance. This limitation primarily arises from significant domain gap between natural images and medical images. To bridge this gap, we introduce SAM-Med2D, the most comprehensive studies on applying SAM to medical 2D images. Specifically, we first collect and curate approximately 4.6M images and 19.7M masks from public and private datasets, constructing a large-scale medical image segmentation dataset encompassing various modalities and objects. Then, we comprehensively fine-tune SAM on this dataset and turn it into SAM-Med2D. Unlike previous methods that only adopt bounding box or point prompts as interactive segmentation approach, we adapt SAM to medical image segmentation through more comprehensive prompts involving bounding boxes, points, and masks. We additionally fine-tune the encoder and decoder of the original SAM to obtain a well-performed SAM-Med2D, leading to the most comprehensive fine-tuning strategies to date. Finally, we conducted a comprehensive evaluation and analysis to investigate the performance of SAM-Med2D in medical image segmentation across various modalities, anatomical structures, and organs. Concurrently, we validated the generalization capability of SAM-Med2D on 9 datasets from MICCAI 2023 challenge. Overall, our approach demonstrated significantly superior performance and generalization capability compared to SAM.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2308.16184

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 42

  1. [1][1] Cheng J, Tian S, Yu L, et al. ResGANet: Residual group attention network for medical image classification and segmentation[J]. Medical Image Analysis, 2022, 76: 102313.
  2. [2][2] Aljabri M, AlGhamdi M. A review on the use of deep learning for medical images segmentation[J]. Neurocomputing, 2022.
  3. [3][3] Xie X, Pan X, Shao F, et al. Mci-net: multi-scale context integrated network for liver ct image segmentation[J]. Computers and Electrical Engineering, 2022, 101: 108085.
  4. [4][4] Ma J, Zhang Y, Gu S, et al. Fast and low-GPU-memory abdomen CT organ segmentation: the flare challenge[J]. Medical Image Analysis, 2022, 82: 102616.
  5. [5][5] Cheng J, Gao C, Wang F, et al. SegNetr: Rethinking the local-global interactions and skip connections in U-shaped networks[J]. arXiv preprint arXiv:2307.02953, 2023. · arXiv
  6. [6][6] Caron M, Touvron H, Misra I, et al. Emerging properties in self-supervised vision transform- ers[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 9650- 9660.
  7. [7][7] Wang X, Zhang X, Cao Y, et al. Seggpt: Segmenting everything in context[J]. arXiv preprint arXiv:2304.03284, 2023. · arXiv
  8. [8][8] Kirillov A, Mintun E, Ravi N, et al. Segment anything[J]. arXiv preprint arXiv:2304.02643, 2023. · arXiv
  9. [9][9] Tang L, Xiao H, Li B. Can sam segment anything? when sam meets camouflaged object detection[J]. arXiv preprint arXiv:2304.04709, 2023. · arXiv
  10. [10][10] Ji G P, Fan D P, Xu P, et al. SAM Struggles in Concealed Scenes–Empirical Study on" Segment Anything"[J]. arXiv preprint arXiv:2304.06022, 2023. · arXiv
  11. [11][11] He S, Bao R, Li J, et al. Accuracy of segment-anything model (sam) in medical image segmen- tation tasks[J]. arXiv preprint arXiv:2304.09324, 2023. · arXiv
  12. [12][12] Zhou T, Zhang Y, Zhou Y, et al. Can sam segment polyps?[J]. arXiv preprint arXiv:2304.07583, 2023. 14 http://arxiv.org/abs/2307.02953 http://arxiv.org/abs/2304.03284 http://arxiv.org/abs/2304.02643 http://arxiv.org/abs/2304.04709 http://arxiv.org/abs/2304.06022 http://arxiv.org/abs/2304.09324 http://arxiv.org/abs/2304.07583 · arXiv
  13. [13][13] Putz F, Grigo J, Weissmann T, et al. The Segment Anything foundation model achieves favorable brain tumor autosegmentation accuracy on MRI to support radiotherapy treatment planning[J]. arXiv preprint arXiv:2304.07875, 2023. · arXiv
  14. [15][15] Huang Y, Yang X, Liu L, et al. Segment anything model for medical images?[J]. arXiv preprint arXiv:2304.14660, 2023. · arXiv
  15. [16][16] Deng G, Zou K, Ren K, et al. SAM-U: Multi-box prompts triggered uncertainty estimation for reliable SAM in medical image[J]. arXiv preprint arXiv:2307.04973, 2023. · arXiv
  16. [17][17] Zhang K, Liu D. Customized segment anything model for medical image segmentation[J]. arXiv preprint arXiv:2304.13785, 2023. · arXiv
  17. [18][18] Hu X, Xu X, Shi Y. How to Efficiently Adapt Large Segmentation Model (SAM) to Medical Images[J]. arXiv preprint arXiv:2306.13731, 2023. · arXiv
  18. [19][19] Bernard O, Lalande A, Zotti C, et al. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved?[J]. IEEE transactions on medical imaging, 2018, 37(11): 2514-2525.
  19. [20][20] Ma J, Wang B. Segment anything in medical images[J]. arXiv preprint arXiv:2304.12306, 2023. · arXiv
  20. [21][21] Wu J, Fu R, Fang H, et al. Medical sam adapter: Adapting segment anything model for medical image segmentation[J]. arXiv preprint arXiv:2304.12620, 2023. · arXiv
  21. [22][22] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint arXiv:2010.11929, 2020. · arXiv
  22. [23][23] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//International conference on machine learning. PMLR, 2021: 8748-8763.
  23. [24][24] Jia C, Yang Y, Xia Y, et al. Scaling up visual and vision-language representation learning with noisy text supervision[C]//International conference on machine learning. PMLR, 2021: 4904-4916.
  24. [25][25] Ramesh A, Pavlov M, Goh G, et al. Zero-shot text-to-image generation[C]//International Conference on Machine Learning. PMLR, 2021: 8821-8831.
  25. [26][26] Zou X, Yang J, Zhang H, et al. Segment everything everywhere all at once[J]. arXiv preprint arXiv:2304.06718, 2023. · arXiv
  26. [27][27] Deng R, Cui C, Liu Q, et al. Segment anything model (sam) for digital pathology: Assess zero-shot segmentation on whole slide imaging[J]. arXiv preprint arXiv:2304.04155, 2023. · arXiv
  27. [28][28] Hu C, Li X. When sam meets medical images: An investigation of segment anything model (sam) on multi-phase liver tumor segmentation[J]. arXiv preprint arXiv:2304.08506, 2023. · arXiv
  28. [29][29] Cheng D, Qin Z, Jiang Z, et al. Sam on medical images: A comprehensive study on three prompt modes[J]. arXiv preprint arXiv:2305.00035, 2023. · arXiv
  29. [30][30] Liu Q, Xu Z, Bertasius G, et al. SimpleClick: Interactive image segmentation with simple vision transformers[J]. arXiv preprint arXiv:2210.11006, 2022. · arXiv
  30. [31][31] Lin T Y, Goyal P, Girshick R, et al. Focal loss for dense object detection[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2980-2988.
  31. [32][32] Milletari F, Navab N, Ahmadi S A. V-net: Fully convolutional neural networks for volumetric medical image segmentation[C]//2016 fourth international conference on 3D vision (3DV). Ieee, 2016: 565-571.
  32. [33][33] Shapey J, Kujawa A, Dorent R, et al. Segmentation of vestibular schwannoma from MRI, an open annotated dataset and baseline algorithm[J]. Scientific Data, 2021, 8(1): 286. 15 http://arxiv.org/abs/2304.07875 http://arxiv.org/abs/2304.14660 http://arxiv.org/abs/2307.04973 http://arxiv.org/abs/2304.13785 http://arxiv.org/abs/2306.13731 http://arxiv.org/abs/2304.12306 http://arxiv.org/abs/2304.12620 http://arxiv.org/abs/2010.11929 http://arxiv.org/abs/2304.06718 http://arxiv.org/abs/2304.04155 http://arxiv.org/abs/2304.08506 http://arxiv.org/abs/2305.00035 http://arxiv.org/abs/2210.11006 · arXiv
  33. [34][34] Heller N, Isensee F, Trofimova D, et al. The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CT[J]. arXiv preprint arXiv:2307.01984, 2023. · arXiv
  34. [35][35] Fast and Low-Resource Semi-supervised Abdominal Organ Segmentation: MICCAI 2022 Challenge, FLARE 2022, Held in Conjunction with MICCAI 2022, Singapore, September 22, 2022, Proceedings[M]. Springer Nature, 2023.
  35. [36][36] Félix Quinton, Sarah Leclerc, Benoît Presles, Fabrice Meriaudeau, Arnaud Boucher, Dominique Ginhac, Huazhu Fu, Xinxing Xu, Jean-Louis Alberini, Romain Popoff" Jean-Marc Vrigneaud, & Olivier Chevalier. (2023). A tumor and liver automatic segmentation challenge. International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 2023.
  36. [37][37] Xiangde Luo, Wenjun Liao, Mu Zhou, Jia Fu, Shichuan Zhang, Guotai Wang, & Shaoting Zhang. (2023). Segmentation of Organs-at-Risk and Gross Tumor Volume for Radiotherapy Planning of Nasopharyngeal Carcinoma Challenge 2023. International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 2023.
  37. [38][38] Ron Kikinis, Steve Pieper, Erik Ziegler, Gordon Harris, Tagwa Idris, Bhanusupriya Somarouthu, Heather Jacene, Roya Khajavi, Reuben Dorent, & Sonia Pujol. (2023). Mediastinal Lymph Node Quantification (LNQ): Segmentation of Heterogeneous CT Data. International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 2023.
  38. [39][39] Huijun Chen, Xihai Zhao, Haozhong Sun, Jiaqi Dou, Chenlin Du, Runyu Yang, Xiaoqi Lin, Shuwan Yu, Jiachen Liu, Chun Yuan, & Niranjan Balu. (2023). Cerebral artery segmentation Challenge. International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 2023.
  39. [40][40] Kuanquan Wang, Mingwang Xu, Qiucheng Wang, Wen Cheng, Wei Wang, & Xinjie Liang. (2022). Tumor Detection, Segmentation, and Classification Challenge on Automated 3D Breast Ultrasound. 26th International Conference on Medical Image Computing and Computer Assisted Intervention MICCAI 2023.
  40. [41][41] Federico Bolelli, Luca Lumetti, Mattia Di Bartolomeo, Shankeeth Vinayahalingam, Alexandre Anesi, Bram van Ginneken, & CostantinoGrana. (2023). Tooth Fairy: A Cone-Beam Computed Tomography Segmentation Challenge. International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) 2023.
  41. [42][42] Kuznetsova A, Rom H, Alldrin N, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale[J]. International Journal of Computer Vision, 2020, 128(7): 1956-1981.
  42. [43][43] Chen Z, Duan Y, Wang W, et al. Vision transformer adapter for dense predictions[J]. arXiv preprint arXiv:2205.08534, 2022. 16 http://arxiv.org/abs/2307.01984 http://arxiv.org/abs/2205.08534 · arXiv

Cited by

Sorted by citations

Authors
Cheng, J., Ye, J., Deng, Z., Chen, J., Li, T., Wang, H., Su, Y., Huang, Z., Chen, J., Jiang, L., Sun, H., He, J., Zhang, S., Zhu, M., & Qiao, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2308.16184
arXiv
2308.16184
OpenAlex
W4386370554
Cited by
26
References
0
Citations by year
2026: 12 · 2025: 4 · 2024: 9 · 2023: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
Segment Anything Model, medical image segmentation, interactive segmentation, prompt-based segmentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review