Computer-Vision Benchmark Segment-Anything Model (SAM) in Medical Images: Accuracy in 12 Datasets
Open accessAbstract
Background: The segment-anything model (SAM), introduced in April 2023, shows promise as a benchmark model and a universal solution to segment various natural images. It comes without previously-required re-training or fine-tuning specific to each new dataset. Purpose: To test SAM's accuracy in various medical image segmentation tasks and investigate potential factors that may affect its accuracy in medical images. Methods: SAM was tested on 12 public medical image segmentation datasets involving 7,451 subjects. The accuracy was measured by the Dice overlap between the algorithm-segmented and ground-truth masks. SAM was compared with five state-of-the-art algorithms specifically designed for medical image segmentation tasks. Associations of SAM's accuracy with six factors were computed, independently and jointly, including segmentation difficulties as measured by segmentation ability score and by Dice overlap in U-Net, image dimension, size of the target region, image modality, and contrast. Results: The Dice overlaps from SAM were significantly lower than the five medical-image-based algorithms in all 12 medical image segmentation datasets, by a margin of 0.1-0.5 and even 0.6-0.7 Dice. SAM-Semantic was significantly associated with medical image segmentation difficulty and the image modality, and SAM-Point and SAM-Box were significantly associated with image segmentation difficulty, image dimension, target region size, and target-vs-background contrast. All these 3 variations of SAM were more accurate in 2D medical images, larger target region sizes, easier cases with a higher Segmentation Ability score and higher U-Net Dice, and higher foreground-background contrast.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021) · arXiv · doi:10.48550/arxiv.2102.04306 · Cited by 4,206 · see citing works
Codella, N., Rotemberg, V., Tschandl, P., Celebi, M. E., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1902.03368 · Cited by 984 · see citing works
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.02643 · Cited by 527 · see citing works
Deng, R., Cui, C., Liu, Q., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.04155 · Cited by 95 · see citing works
Segment Anything in Medical Images
Ma, J., He, Y., Li, F., Han, L., You, C., & Wang, B. (2023) · arXiv · doi:10.48550/arxiv.2304.12306 · Cited by 67 · see citing works
SAM on Medical Images: A Comprehensive Study on Three Prompt ModesPreprint
Cheng, D., Qin, Z., Jiang, Z., Zhang, S., Lao, Q., & Li, K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.00035 · Cited by 66 · see citing works
Hu, C., Xia, T., Ju, S., & Li, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.08506 · Cited by 43 · see citing works
Chen, T., Zhu, L., Ding, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.09148 · Cited by 42 · see citing works
Can SAM Segment Polyps?Preprint
Zhou, T., Zhang, Y., Zhou, Y., Wu, Y., & Gong, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.07583 · Cited by 28 · see citing works
U-Netmer: U-Net meets Transformer for medical image segmentationPreprint
He, S., Bao, R., Grant, P. E., & Ou, Y.-M. (2023) · arXiv.org · doi:10.48550/arxiv.2304.01401 · Cited by 25 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 25
- O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Pro- ceedings, Part III 18. Springer, 2015, pp. 234-241.
- Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, "Unet++: Redesigning skip connections to exploit multiscale features in image segmentation," IEEE transactions on medical imaging, vol. 39, no. 6, pp. 1856-1867, 2019.
- M. H. Hesamian, W. Jia, X. He, and P. Kennedy, "Deep learning techniques for medical image segmentation: achievements and challenges," Journal of digital imaging, vol. 32, pp. 582-596, 2019.
- X. Liu, L. Song, S. Liu, and Y. Zhang, "A review of deep- learning-based medical image segmentation methods," Sustain- ability, vol. 13, no. 3, p. 1224, 2021.
- S. K. Zhou, H. Greenspan, C. Davatzikos, J. S. Duncan, B. Van Gin- neken, A. Madabhushi, J. L. Prince, D. Rueckert, and R. M. Sum- mers, "A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises," Proceedings of the IEEE, vol. 109, no. 5, pp. 820-838, 2021.
- O. Bernard, A. Lalande, C. Zotti, F. Cervenansky, X. Yang, P.- A. Heng, I. Cetin, K. Lekadir, O. Camara, M. A. G. Ballester et al., "Deep learning techniques for automatic mri cardiac multi- structures segmentation and diagnosis: is the problem solved?" IEEE transactions on medical imaging, vol. 37, no. 11, pp. 2514-2525, 2018.
- B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest et al., "The mul- timodal brain tumor image segmentation benchmark (BRATS)," IEEE transactions on medical imaging, vol. 34, no. 10, pp. 1993-2024, 2014.
- S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Farahani, and C. Davatzikos, "Advancing the cancer genome atlas glioma mri collections with expert segmenta- tion labels and radiomic features," Scientific data, vol. 4, no. 1, pp. 1-13, 2017.
- S. Bakas, M. Reyes, A. Jakab, S. Bauer, M. Rempfler, A. Crimi, R. T. Shinohara, C. Berger, S. M. Ha, M. Rozycki et al., "Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge," arXiv preprint arXiv:1811.02629, 2018. · arXiv
- W. Al-Dhabyani, M. Gomaa, H. Khaled, and A. Fahmy, "Dataset of breast ultrasound images," Data in brief, vol. 28, p. 104863, 2020.
- W. Choi, N. Dahiya, and S. Nadeem, "CIRDataset: A large-scale dataset for clinically-interpretable lung nodule radiomics and ma- lignancy prediction," in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2022, pp. 13-22.
- D. Jha, P. H. Smedsrud, M. A. Riegler, P. Halvorsen, T. d. Lange, D. Johansen, and H. D. Johansen, "Kvasir-seg: A segmented polyp dataset," in International Conference on Multimedia Modeling. Springer, 2020, pp. 451-462.
- M. A. Attiyeh, J. Chakraborty, A. Doussot, L. Langdon-Embry, S. Mainarich, M. G önen, V. P. Balachandran, M. I. D'Angelica, R. P. DeMatteo, W. R. Jarnagin et al., "Survival prediction in pancreatic ductal adenocarcinoma by quantitative computed tomography image analysis," Annals of surgical oncology, vol. 25, no. 4, pp. 1034- 1042, 2018.
- M. Antonelli, A. Reinke, S. Bakas, K. Farahani, A. Kopp-Schneider, B. A. Landman, G. Litjens, B. Menze, O. Ronneberger, R. M. Summers et al., "The medical segmentation decathlon," Nature communications, vol. 13, no. 1, pp. 1-13, 2022.
- Q. Liu, Q. Dou, L. Yu, and P. A. Heng, "MS-Net: multi-site network for improving prostate segmentation with heterogeneous mri data," IEEE transactions on medical imaging, vol. 39, no. 9, pp. 2713-2724, 2020.
- P. Tschandl, C. Rosendahl, and H. Kittler, "The ham10000 dataset, a large collection of multi-source dermatoscopic images of com- mon pigmented skin lesions," Scientific data, vol. 5, no. 1, pp. 1-9, 2018.
- Z. Xiong, Q. Xia, Z. Hu, N. Huang, C. Bian, Y. Zheng, S. Vesal, N. Ravikumar, A. Maier, X. Yang et al., "A global benchmark of algorithms for segmenting the left atrium from late gadolinium- enhanced cardiac magnetic resonance imaging," Medical image analysis, vol. 67, p. 101832, 2021.
- P. Bilic, P. Christ, H. B. Li, E. Vorontsov, A. Ben-Cohen, G. Kaissis, A. Szeskin, C. Jacobs, G. E. H. Mamani, G. Chartrand et al., "The liver tumor segmentation benchmark (LiTS)," Medical Image Analysis, vol. 84, p. 102680, 2023.
- S. Jaeger, S. Candemir, S. Antani, Y.-X. J. Wáng, P.-X. Lu, and G. Thoma, "Two public chest x-ray datasets for computer-aided screening of pulmonary diseases," Quantitative imaging in medicine and surgery, vol. 4, no. 6, p. 475, 2014.
- J. Schlemper, O. Oktay, M. Schaap, M. Heinrich, B. Kainz, B. Glocker, and D. Rueckert, "Attention gated networks: Learning to leverage salient regions in medical images," Medical image analysis, vol. 53, pp. 197-207, 2019.
- H. Wang, P. Cao, J. Wang, and O. R. Zaiane, "Uctransnet: rethink- ing the skip connections in u-net from a channel-wise perspective with transformer," in Proceedings of the AAAI conference on artificial intelligence, vol. 36, no. 3, 2022, pp. 2441-2449.
- S. He, Y. Feng, P. E. Grant, and Y. Ou, "Segmentation ability map: Interpret deep features for medical image segmentation," Medical Image Analysis, vol. 84, p. 102726, 2023.
- E. Peli, "Contrast in complex images," JOSA A, vol. 7, no. 10, pp. 2032-2040, 1990.
- V. Remeš and M. Haindl, "Region of interest contrast measures," Kybernetika, vol. 54, no. 5, pp. 978-990, 2018.
- S. Mohapatra, A. Gosai, and G. Schlaug, "Brain extraction compar- ing segment anything model (sam) and fsl brain extraction tool," arXiv preprint arXiv:2304.04738, 2023. · arXiv
Cited by
Sorted by citations
Segment anything in medical images
Ma, J., He, Y., Li, F., Han, L., You, C., & Wang, B. (2024) · Nature Communications · doi:10.1038/s41467-024-44824-z · Cited by 2,913 · see citing works
Segment anything model for medical image analysis: An experimental study
Mazurowski, M. A., Dong, H., Gu, H., Yang, J., Konz, N., & Zhang, Y. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102918 · Cited by 802 · see citing works
Segment anything model for medical images?
Huang, Y., Yang, X., Liu, L., Zhou, H., Chang, A., Zhou, X., Chen, R., Yu, J., Chen, J., Chen, C., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2023.103061 · Cited by 536 · see citing works
Segment Anything in Medical Images
Ma, J., He, Y., Li, F., Han, L., You, C., & Wang, B. (2023) · arXiv · doi:10.48550/arxiv.2304.12306 · Cited by 67 · see citing works
- Authors
- He, S., Bao, R., Li, J., Stout, J., Bjornerud, A., Grant, P. E., & Ou, Y.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.09324
- OpenAlex
- W4366552071
- Cited by
- 59
- References
- 10
- Citations by year
- 2026: 2 · 2025: 10 · 2024: 34 · 2023: 13
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- Segment Anything Model, medical image segmentation, imaging modalities, image dimensions, U-Net comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
