CoChat
Sign Up
Cited by 536Medical Image Segmentation TechniquesAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

Segment anything model for medical images?

Open access
Huang, Yuhao, Yang, Xin, Liu, Lian, Zhou, Han, Chang, Ao, Zhou, Xinrui, Chen, Rusi, Yu, Junxuan +11 more·2024·Medical Image Analysis
DOI 10.1016/j.media.2023.103061
Open this paper in CoChat — free accountOpen PDF

Abstract

The Segment Anything Model (SAM) is the first foundation model for general image segmentation. It has achieved impressive results on various natural image segmentation tasks. However, medical image segmentation (MIS) is more challenging because of the complex modalities, fine anatomical structures, uncertain and complex object boundaries, and wide-range object scales. To fully validate SAM's performance on medical data, we collected and sorted 53 open-source datasets and built a large medical segmentation dataset with 18 modalities, 84 objects, 125 object-modality paired targets, 1050K 2D images, and 6033K masks. We comprehensively analyzed different models and strategies on the so-called COSMOS 1050K dataset. Our findings mainly include the following: (1) SAM showed remarkable performance in some specific objects but was unstable, imperfect, or even totally failed in other situations. (2) SAM with the large ViT-H showed better overall performance than that with the small ViT-B. (3) SAM performed better with manual hints, especially box, than the Everything mode. (4) SAM could help human annotation with high labeling quality and less time. (5) SAM was sensitive to the randomness in the center point and tight box prompts, and may suffer from a serious performance drop. (6) SAM performed better than interactive methods with one or a few points, but will be outpaced as the number of points increases. (7) SAM's performance correlated to different factors, including boundary complexity, intensity differences, etc. (8) Finetuning the SAM on specific medical tasks could improve its average DICE performance by 4.39% and 6.68% for ViT-B and ViT-H, respectively. Codes and models are available at: https://github.com/yuhoo0302/Segment-Anything-Model-for-Medical-Images. We hope that this comprehensive report can help researchers explore the potential of SAM applications in MIS, and guide how to appropriately use and develop SAM.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper builds COSMOS 1050K, a large medical segmentation benchmark from 53 public datasets, and comprehensively evaluates the Segment Anything Model (SAM) across modalities, objects, prompt strategies, and model sizes. SAM is highly variable in medical image segmentation—strong on some targets but unstable or failing on others—while box prompts, larger ViT-H, and task-specific finetuning improve performance.

Key findings

  • SAM’s performance was unstable across medical objects: remarkable for some, but imperfect or totally failed in others. Manual prompts outperformed Everything mode, and box prompts were best; adding a point to a box did not obviously help (ViT-B DICE 0.49% ↓, ViT-H DICE 0.06% ↓). Increasing point prompts improved DICE from 56.02% to 63.81% for ViT-B and from 56.78% to 71.61% for ViT-H.
  • ViT-H generally outperformed ViT-B. Under Everything mode, ViT-H exceeded ViT-B by 7.47% DICE and had 10.61 pixels lower HD. Modality-wise, SAM achieved the highest average DICE on X-ray for both models; histopathology, colonoscopy, and DW MRI also reached >90% DICE, while 6 (ViT-B) and 7 (ViT-H) modalities had DICE between 80 and 90 but with large standard deviations.
  • SAM was sensitive to prompt randomness: for one positive point, DICE dropped by 2.67%, 7.38%, and 14.62% as shifting increased; for one box, shifts of 20–30 pixels caused a 24.11% decrease, and one box plus one point caused a 29.93% decrease. In interactive comparison, SAM beat FocalClick and SimpleClick with one or a few points but was outpaced as points increased, performing worse at 10 points.
  • Finetuning SAM on specific medical tasks improved average DICE by 4.39% for ViT-B and 6.68% for ViT-H; 32/45 objects improved for ViT-B and 37/45 for ViT-H. DICE showed moderate correlation with Fourier order and intensity difference, weak correlation with size, and no correlation with modality or aspect ratio.

Methods

  • The authors collected and standardized 53 public datasets into COSMOS 1050K: 18 modalities, 84 objects, 125 object-modality paired targets, 1,050,311 2D images/slices (1,003,809 slices from 8,653 3D volumes and 46,502 standalone 2D images), and 6,033,198 masks.
  • They evaluated SAM with ViT-B (91M parameters, 12 transformer layers) and ViT-H (636M parameters, 32 transformer layers) under six strategies: S1 Everything; S2 one positive point; S3 five positive points; S4 five positive and five negative points; S5 one box; S6 one box and one positive point. Metrics were DICE, JAC, and HD, with a mask-matching mechanism; Spearman partial correlation analyzed 191,779 anatomical structures for size, aspect ratio, intensity difference, modality, and boundary complexity via Elliptical Fourier Descriptors/Fourier order.
  • Finetuning used 45 common objects with box prompts, frozen image and prompt encoders, and only the mask decoder adjusted for 20 epochs at learning rate 1e-4 and batch size 2. A separate annotation study involved three doctors, 100 images, 55 objects, 620 masks, and 9 modalities, and SAM was compared with FocalClick and SimpleClick.

Limitations

  • The paper states that SAM is unstable in medical image segmentation, has weak point-based multi-round iteration capability, is sensitive to prompt randomness, cannot output object categories/semantics, and was trained on natural images without medical data. It also notes that finetuning decreased performance for objects with small numbers, RGB color coding, etc., requiring more careful task-specific finetuning.
  • Evident from the text: the evaluation is 2D-only, with 3D/4D data converted to 2D slices; the dataset excludes extremely small objects, objects that become significantly separated across slices, and discrete structures such as some histopathology and lung trachea trees.
  • Evident from the text: the main paper presents detailed performance for only 40 objects, with full results in supplementary materials, and the text notes that SAM’s text prompt had not been released in the official repository at the time.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.media.2023.103061

Cites

104 of 104 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 31 cited sources checked · 0 supported · 0 flagged

  • Findingown data
    § A B S T R A C T · p. 1

    The authors built a large medical segmentation dataset (COSMOS 1050K) with 18 modalities.

    “we collected and sorted 53 open-source datasets and built a large medical segmentation dataset with 18 modalities, 84 objects, 125 object-modality paired targets, 1050K 2D images, and 6033K masks.”
    • Also stated in: § 1.2. SAM in Medical Image Analysis · p. 5 · plain
      “we build a large medical image dataset named COSMOS 1050K , including 1050 K images with 18 different modalities (see Fig. 1) and 84 objects ( e.g., anatomical structures, lesions, cells, tools, etc.), to cover the entire body (see Fig. 2).”
    Quote in the paperSubstantial claim
  • Findingown data
    § A B S T R A C T · p. 1

    COSMOS 1050K consists of 1,050,311 2D images or slices.

    “we collected and sorted 53 open-source datasets and built a large medical segmentation dataset with 18 modalities, 84 objects, 125 object-modality paired targets, 1050K 2D images, and 6033K masks.”
    • Also stated in: § 2.2. Dataset Preprocessing Specification · p. 7 · plain
      “In total, COSMOS 1050K consists of 1,050,311 2D images or slices, with 1,003,809 slices originating from 8,653 3D volumes and 46,502 being standalone 2D images.”
    Quote in the paperNot a substantial claim
  • Findingnone given
    § A B S T R A C T · p. 1

    SAM was sensitive to the randomness in the center point and tight box prompts.

    “SAM was sensitive to the randomness in the center point and tight box prompts, and may suffer from a serious performance drop.”
    Quote in the paperSubstantial claim
  • Findingown data
    § A B S T R A C T · p. 1

    SAM performed better than interactive methods with one or a few points.

    “SAM performed better than interactive methods with one or a few points, but will be outpaced as the number of points increases.”
    Quote in the paperSubstantial claim
  • Findingown data
    § A B S T R A C T · p. 1

    SAM performed better with manual hints, especially box, than the Everything mode.

    “SAM performed better with manual hints, especially box, than the Everything mode.”
    • Also stated in: § 5. Conclusion · p. 19 · plain
      “3) SAM performed better with manual hints, especially box, than the Everything mode.”
    Quote in the paperSubstantial claim
  • May combine two claims. 25
  • Classified as probably not a substantial claim. 17
  • The quote only partly states this claim. 1

  • May combine two claims.

    Findingown data
    § A B S T R A C T · p. 1

    SAM was unstable, imperfect, or even totally failed in other situations.

    “SAM showed remarkable performance in some specific objects but was unstable, imperfect, or even totally failed in other situations.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.2. Dataset Preprocessing Specification · p. 7

    Of the 1,050,311 2D images, 46,502 are standalone 2D images.

    “In total, COSMOS 1050K consists of 1,050,311 2D images or slices, with 1,003,809 slices originating from 8,653 3D volumes and 46,502 being standalone 2D images.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 2.2. Dataset Preprocessing Specification · p. 7

    Of the 1,050,311 2D images, 1,003,809 slices originate from 8,653 3D volumes.

    “In total, COSMOS 1050K consists of 1,050,311 2D images or slices, with 1,003,809 slices originating from 8,653 3D volumes and 46,502 being standalone 2D images.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.6. Inference Time Analysis of SAM · p. 11

    For manual prompt encoding and mask decoding (S2-S6), the reference times of different models and strategies are similar and less than 0.01s.

    “While for the manual prompt encoding and mask decoding ( S 2 -S 6), the reference times of different models and strategies are similar and less than 0.01s.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.5. Performance of Medical Images on Different Modalities · p. 10

    There are 6 ViT-B and 7 ViT-H modalities with DICE performance between 80 and 90, but most of them have large standard deviations.

    “There are 6 ( ViTB ) and 7 ( ViT-H ) modalities with DICE performance between 80 and 90, but most of them have large standard deviations.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The ground-truth masks used for evaluation accurately represent the true segmentation targets.
    • Rests on an assumption: The selected datasets and anatomical structures are representative of medical images, so the evaluation results can provide general guidance.

  • May combine two claims.

    Findingown data
    § 4.5. Performance of Medical Images on Different Modalities · p. 10

    BrainTumor performance ranges from 71.76% to 77.74% in DICE and 13.02 to 15.35 pixels in HD.

    “See BrainTumor in Table 2 and 3, its performance ranges from 71.76% to 77.74% in DICE and 13.02 pixels to 15.35 pixels in HD.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The ground-truth masks used for evaluation accurately represent the true segmentation targets.

  • May combine two claims.

    Findingown data
    § 4.5. Performance of Medical Images on Different Modalities · p. 10

    SAM performs worse in the remaining modalities, and the results are also more unstable.

    “SAM performs worse in the remaining modalities, and the results are also more unstable.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The ground-truth masks used for evaluation accurately represent the true segmentation targets.
    • Rests on an assumption: The selected datasets and anatomical structures are representative of medical images, so the evaluation results can provide general guidance.

  • May combine two claims.

    Findingreasoning
    § 3.4. Inference Efficiency · p. 9

    Precomputing and saving image embeddings allows them to be reused, improving overall SAM testing efficiency by nearly n times.

    “Accordingly, the image embeddings could be reused to relieve the computational burden of the inference pipeline. Thereby, the overall efficiency of the SAM testing could be improved by nearly n times.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The image embeddings computed once can be reused across different testing strategies for the same image without affecting segmentation results.

  • Classified as probably not a substantial claim.

    Findingnone given
    § 4.1. Implementation Details · p. 9

    The process of image embedding extraction was time-consuming.

    “We observed that the process of image embedding extraction was time-consuming.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.4. Segmentation Performance under Different Testing Modes · p. 10

    SAM with a box prompt yields the best performance, while adding one point to the box does not bring obvious changes: ViT-B DICE decreases 0.49%, ViT-H DICE decreases 0.06%.

    “SAM with a box prompt yields the best performance, while adding one point to the box will not bring obvious changes (ViT-B: DICE 0.49% ↓ , ViT-H: DICE 0.06% ↓ ).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The ground-truth masks used for evaluation accurately represent the true segmentation targets.

  • May combine two claims.

    Findingnone given
    § 3.5. Mask-matching Mechanism for Segmentation Evaluation · p. 9

    SAM generated multiple binary masks for each input image, but not all of them included the corresponding object.

    “SAM generated multiple binary masks for each input image, but not all of them included the corresponding object.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § 4.1. Implementation Details · p. 9

    Refactoring the code to extract image embeddings once and save them as npy files significantly improved testing efficiency, approximately n times.

    “When different testing strategies were applied, only the corresponding npy files needed to be loaded, which significantly improved the testing efficiency (approximately n × ).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The official GitHub implementation of SAM is correct and faithful.
    • Rests on an assumption: The image embeddings computed once can be reused across different testing strategies for the same image without affecting segmentation results.

  • May combine two claims.

    Findingown data
    § 4.4. Segmentation Performance under Different Testing Modes · p. 10

    For point prompts (S2-S4), adding more points brings stable performance improvement: ViT-B DICE from 56.02% to 63.81%, ViT-H DICE from 56.78% to 71.61%.

    “For the point prompts ( S 2 -S 4), adding more points will bring stable performance improvement (ViTB: DICE from 56.02% to 63.81%, ViT-H: DICE from 56.78% to 71.61%).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The ground-truth masks used for evaluation accurately represent the true segmentation targets.

  • May combine two claims.

    Findingown data
    § 4.3. Segmentation Performance under Different Models · p. 10

    Under Everything mode (S1), ViT-H exceeds ViT-B by 7.47% in DICE and is 10.61 pixels lower than ViT-B on HD.

    “It can be observed from Fig. 6 that, under Everything mode ( S 1 ), ViT-H exceeds ViT-B with 7.47% at DICE and is 10.61 pixels lower than ViT-B on HD.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The ground-truth masks used for evaluation accurately represent the true segmentation targets.

  • May combine two claims.

    Findingnone given
    § 4.12. Task-specific Refinement for SAM · p. 19

    The performance of objects with small numbers, RGB color coding, etc., decreases after finetuning.

    “The performance of objects with small numbers, RGB color coding, etc., decreases.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § 4.10. Impact of Different Prompt Randomness on Performance · p. 17

    For S2 (single point), the DICE performance dropped by 2.67%, 7.38%, and 14.62% as the shifting level increased (0-10, 10-20, 20-30 pixels).

    “For S 2 (single point), the DICE performance dropped by 2.67%, 7.38%, and 14.62% as the shifting level increased.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given
    § 6. Discussion · p. 20

    SAM can greatly shorten the annotation time and improve the annotation quality.

    “Our study also preliminarily verified that SAM can greatly shorten the annotation time and improve the annotation quality.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.10. Impact of Different Prompt Randomness on Performance · p. 17

    With the increase in the number of point prompts (S3 and S4), the decline of DICE could be alleviated, and the stability of the model could be improved.

    “With the increase in the number of point prompts ( S 3 and S 4), the decline of DICE could be alleviated, and the stability of the model could be improved.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.11. Comparison between SAM and Interactive Methods · p. 19

    As iterations progressed, SAM's performance increased slowly, or even declined, while the performance of the interactive method could be improved steadily.

    “Based on the DICE results, our conclusion is that: 1) SAM outperformed FocalClick and SimpleClick in the first interaction with a single point; 2) As iterations progressed, SAM's performance increased slowly, or even declined, while the performance of the interactive method could be improved steadily; 3) Using 10 points, SAM performed worse than the interactive methods.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The selected 10 typical organs/tumors are representative of medical segmentation tasks for comparing SAM with interactive methods.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.3. Manual Prompt Mode · p. 8

    The text prompt has not been released on the official GitHub repository.

    “Notably, the text prompt has not been released yet on the official GitHub repository 4 .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4] Bernal, 2015, WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians, Comput. Med. Imaging Graph. source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning
    § 4.6. Inference Time Analysis of SAM · p. 11

    The study conducted a fair comparison by limiting image size to 256 × 256 and setting the number of targets to 1.

    “Thus, we conducted a fair comparison by limiting the image size to 256 × 256 and setting the number of targets to 1.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given
    § 2.2. Dataset Preprocessing Specification · p. 6

    Retaining slices with the sum of the pixel values of their labels greater than 50 ensures that each slice has the corresponding correct label.

    “Retain slices with the sum of the pixel values of their labels greater than 50 for any 3D image and label volumes. This ensures that each slice has the corresponding correct label.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4.4. Segmentation Performance under Different Testing Modes · p. 10

    A box tells the exact location of the target and its potential intensity features given the limited region, whereas points only represent part features of the target, which can lead to confusion.

    “Since the box actually tells the exact location of the target and also its potential intensity features given the limited region. However, points only represent the part features of the target, which can lead to confusion.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The ground-truth masks used for evaluation accurately represent the true segmentation targets.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.8. Analysis of Factors Correlating to Segmentation Results · p. 14

    As the Fourier order (FO) increases, the contour decoded from the Fourier series gets closer to the original contour.

    “As the Fourier order (FO) increases, the contour decoded from the Fourier series gets closer to the original contour (see Fig. 12)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kuhl and Giardina, 1982

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 3.1. Brief Introduction to SAM · p. 8

    A study using six medical datasets found no obvious winner among ViT-B, ViT-L, and ViT-H.

    “While evaluating SAM in medical images, one study used six medical datasets and found that there is no obvious winner among ViT-B , ViT-L , and ViT-H (Mattjie et al., 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Mattjie et al., 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.1. Brief Introduction to SAM · p. 8

    In Kirillov et al. (2023), ViT-H shows substantial performance improvements over ViT-B, but requires multiplied testing time due to increased complexity.

    “In Kirillov et al. (2023), ViT-H shows substantial performance improvements over ViT-B . However, the former requires multiplied testing time due to its increased complexity.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kirillov et al. (2023)

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning
    § 4.3. Segmentation Performance under Different Models · p. 10

    Compared to point prompts, box prompts contain more region information about the object and thus better guide SAM with different models to achieve better segmentation performance.

    “Compared to the point prompt, the box prompt contains more region information about the object. Thus, it can better guide SAM with different models to achieve better segmentation performance.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The ground-truth masks used for evaluation accurately represent the true segmentation targets.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.1. Brief Introduction to SAM · p. 8

    SAM can only find multiple targets in an image without outputting their detailed categories; it is single-label, object or not.

    “It is noted that SAM can only find multiple targets in the image, without outputting their detailed categories (i.e., single-label: object or not).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kirillov et al. (2023)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 3.1. Brief Introduction to SAM · p. 7

    SAM introduces a promptable segmentation task supported by a flexible prompting-enabled model architecture and vast and diverse training data.

    “SAM diverges from traditional segmentation frameworks by introducing a novel promptable segmentation task , which is supported by a flexible prompting-enabled model architecture and vast and diverse sources of training data .”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kirillov et al. (2023)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 3.3. Manual Prompt Mode · p. 8

    The point-selection strategy ensures repeatability of the experiments.

    “The above strategy can ensure the repeatability of the experiments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given
    § 3.3. Manual Prompt Mode · p. 8

    The unified rule for point selection ensures randomness, repeatability, and accuracy.

    “We further established a unified rule for point selection to ensure randomness, repeatability, and accuracy.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 3.3. Manual Prompt Mode · p. 8

    For fair comparison, the study tests one-round interaction performance of SAM under five prompt strategies.

    “Thus, for a fair comparison, we test the one-round interaction performance of SAM under the above five prompt strategies ( S 2 -S 6 ).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 3.4. Inference Efficiency · p. 9

    The embedding-reuse optimization can be extended to other multi-strategy testing scenarios of SAM.

    “This can simply be extended to other multi-strategy testing scenarios of SAM.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The image embeddings computed once can be reused across different testing strategies for the same image without affecting segmentation results.

  • May combine two claims.

    Interpretationown data
    § 5. Conclusion · p. 19

    SAM showed remarkable performance in some specific objects but was unstable, imperfect, or even totally failed in other situations.

    “1) SAM showed remarkable performance in some specific objects but was unstable, imperfect, or even totally failed in other situations.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § 6. Discussion · p. 20

    They shorten the annotation time to three weeks with the help of multiple pre-trained segmentation models for generating pseudo labels, and other useful strategies.

    “They shorten the annotation time to three weeks with the help of multiple pre-trained segmentation models for generating pseudo labels, and other useful strategies.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Qu et al., 2023

      Not checked in the free run.

  • May combine two claims.

    Premisenone given
    § 4.12. Task-specific Refinement for SAM · p. 19

    Most medical images are gray-scale, with unclear and complex object boundaries, similar back- and fore-ground, and wide-range image size (especially containing some very small objects).

    “However, most medical images are gray-scale, with unclear and complex object boundaries, similar back- and fore-ground, and wide-range image size (especially containing some very small objects).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 4.12. Task-specific Refinement for SAM · p. 19

    The training dataset of SAM, SA-1B, contains 11 million photos, including natural locations, objects, and scenes, but without any medical images.

    “The training dataset of SAM, i.e., SA-1B 5 contains 11 million photos, including natural locations, objects, and scenes, but without any medical images.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [5] Bernard, 2018, Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved?, IEEE Trans. Med. Imaging source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning
    § 6. Discussion · p. 20

    A single SAM network can be used to achieve the annotation of large-scale multi-modal, multi-class medical datasets, rather than using multiple task-specific models.

    “It means that a single SAM network can be used to achieve the annotation of large-scale multi-modal, multi-class medical datasets, rather than using multiple task-specific models.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 5. Conclusion · p. 19

    Although SAM has the potential to become a general MIS model, its performance in the MIS task is not stable at present.

    “Finally, we believe that, although SAM has the potential to become a general MIS model, its performance in the MIS task is not stable at present.”
    Quote in the paperSubstantial claim
39 of 39

  • The paper relies on this without support.

    Different medical objects often exhibit different characteristics and thus may have their own suitable modes for testing.

    “However, different medical objects often exhibit different characteristics and thus may have their own suitable modes for testing.”

  • The paper relies on this without support.

    The ground-truth masks used for evaluation accurately represent the true segmentation targets.

    “we first calculated the center of mass of the ground truth (GT) mask”
    “we calculated a set of dice scores { DICE n } N n = 1 between N binary predicted masks { P n } N n = 1 and the GT G .”

  • The paper relies on this without support.

    The selected datasets and anatomical structures are representative of medical images, so the evaluation results can provide general guidance.

    “We hope that a comprehensive evaluation of models of different sizes on large COSMOS 1050K can provide more inspirations for researchers.”
    “we recorded the size, aspect ratio, intensity difference between foreground and background, modality, and boundary complexity of 191,779 anatomical structures.”

  • The paper relies on this without support.

    The official GitHub implementation of SAM is correct and faithful.

    “we implemented the testing pipeline of SAM basically following the official GitHub repository 4 .”

  • The paper relies on this without support.

    The image embeddings computed once can be reused across different testing strategies for the same image without affecting segmentation results.

    “since the same embedding could be reused for different testing strategies, we sought to optimize and accelerate this multi-extraction process.”

  • The paper relies on this without support.

    The center of mass and tight bounding box of the ground truth represent the theoretical optimum prompts for SAM.

    “Besides, we tend to test the theoretical optimum performance of SAM via the selection of the center of mass and tight box. Since they may include the most representative features of the target.”

  • The paper relies on this without support.

    The 100 images randomly sampled from COSMOS 1050K with average DICE performance are representative of the broader medical image dataset for evaluating annotation time and quality.

    “We randomly sampled 100 images with average DICE performance from COSMOS 1050K , to construct an evaluation subset comprising 55 objects and 620 masks across 9 modalities, including instances of the same object in different modalities.”

  • The paper relies on this without support.

    The three doctors with 10 years of experience are representative of human annotators for evaluating annotation time and quality.

    “We then invited three doctors with 10 years of experience to evaluate whether SAM's prediction under box prompts could improve annotation speed and quality.”

  • The paper relies on this without support.

    The five attributes (size, aspect ratio, intensity difference, modality, Fourier order) adequately capture the factors that affect SAM's segmentation performance.

    “We conducted an analysis of the partial correlation between the five attributes of the target objects mentioned above and the DICE score using Spearman's rank partial correlation coefficient under different testing strategies.”

  • The paper relies on this without support.

    The Spearman partial correlation analysis assumptions (e.g., monotonic relationships, no unmeasured confounders) hold for the data.

    “We conducted an analysis of the partial correlation between the five attributes of the target objects mentioned above and the DICE score using Spearman's rank partial correlation coefficient under different testing strategies.”

  • The paper relies on this without support.

    The selected 10 typical organs/tumors are representative of medical segmentation tasks for comparing SAM with interactive methods.

    “We selected 10 typical organs / tumors, covering various modalities, shapes, sizes, and intensity distributions.”
11 of 11

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SAM achieves satisfactory segmentation outcomes primarily on medical targets with distinct boundaries but has substantial limitations on targets with weak boundaries or low contrast.

    “For example, the concurrent studies22,23 conducted a comprehensive assessment of SAM across a diverse array of medical images, underscoring that SAM achieved satisfactory segmentation outcomes primarily on targets characterized by distinct boundaries. However, the model exhibited substantial limitations in segmenting typical medical targets with weak boundaries or low contrast.”

    From Segment anything in medical images (2024) · cites this paper as [23]

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al., 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In: International Conference on Learning Representations.
  2. Huazhu, 2019, PALM: PAthoLogic myopia challenge, Comput. Vis. Med. Imaging
  3. Lee, S., Shim, H., Park, S.H., Yun, I.D., Lee, S.U., 2010. Learning local shape and appearance for segmentation of knee cartilage in 3D MRI. In: Medical Image Analysis for the Clinic: A Grand Challenge. in Proceedings of the 13th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2010), Beijing, China. pp. 231–240.

Cited by

Sorted by citations

Related

Authors
Huang, Y., Yang, X., Liu, L., Zhou, H., Chang, A., Zhou, X., Chen, R., Yu, J., Chen, J., Chen, C., Liu, S., Chi, H., Hu, X., Yue, K., Li, L., Grau, V., Fan, D.-P., Dong, F., & Ni, D.
Affiliations
  • Shenzhen University (CN)
  • Zhejiang University (CN)
  • Hunan First Normal University (CN)
  • University of Oxford (GB)
  • ETH Zurich (CH)
  • Jinan University (CN)
Corresponding author
  • Dong, Fajin
  • Ni, Dong
ORCID
Venue
Medical Image Analysis
Publisher
Elsevier BV
Published
2024-02
Volume
92
Pages
103061
ISSN
1361-8415 (ISSN-L)
DOI
10.1016/j.media.2023.103061
OpenAlex
W4389430914
Semantic Scholar
057ddecf9b9dedd35f824c8b289c30316b4fa22e
Cited by
536
References
101
Citations by year
2026: 134 · 2025: 230 · 2024: 154 · 2023: 18
FWCI
48.56
Type
journal article
Language
en
Open access
Licence
Locations
  • Medical Image Analysis (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • ePrints Soton (University of Southampton) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Natural Science Foundation of China
  • Shenzhen Science and Technology Innovation Committee
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
Segment Anything Model, medical image segmentation, foundation models, point prompts, interactive segmentation, fine-tuning, boundary complexity
MeSH terms
Diagnostic Imaging, Humans, Image Processing, Computer-Assisted

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review