Promise: Prompt-Driven 3D Medical Image Segmentation Using Pretrained Image Foundation Models
Open accessAbstract
To address prevalent issues in medical imaging, such as data acquisition challenges and label availability, transfer learning from natural to medical image domains serves as a viable strategy to produce reliable segmentation results. However, several existing barriers between domains need to be broken down, including addressing contrast discrepancies, managing anatomical variability, and adapting 2D pretrained models for 3D segmentation tasks. In this paper, we propose ProMISe, a prompt-driven 3D medical image segmentation model using only a single point prompt to leverage knowledge from a pretrained 2D image foundation model. In particular, we use the pretrained vision transformer from the Segment Anything Model (SAM) and integrate lightweight adapters to extract depth-related (3D) spatial context without updating the pretrained weights. For robust results, a hybrid network with complementary encoders is designed, and a boundary-aware loss is proposed to achieve precise boundaries. We evaluate our model on two public datasets for colon and pancreas tumor segmentations, respectively. Compared to the state-of-the-art segmentation methods with and without prompt engineering, our proposed method achieves superior performance. The code is publicly available at https://github.com/MedICL-VU/ProMISe.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most promptable medical segmentation methods are still developed and evaluated around clean-prompt protocols.
“Most promptable medical segmentation methods, however, are still developed and evaluated around clean-prompt protocols.”
From Correction-aware interactive 3D tumor segmentation with sparse and revisable prompts (2026) · cites this paper as [17]
The Methods > Prompt encoder section (page unavailable) contrasts ProMISe’s training with a previous method that used points from the target region, which lends limited support to the idea that some prior work used carefully placed prompts. It does not establish that most promptable medical segmentation methods were developed and evaluated under clean-prompt protocols; ProMISe itself trains with background points to improve generalizability to noisy prompts. (LLM, conf=0.76)
Experimental — accuracy not yet measured3DSAM-Adapter and SAM-Med3D adapt SAM-style prompting to 3D medical images, while ProMISe is a prompt-driven 3D medical segmentation method based on pretrained image foundation models.
“3DSAM-Adapter [5] and SAM-Med3D [29] adapt SAM-style prompting to 3D medical images, while ProMISe [17] is a prompt-driven 3D medical segmenta-tion method based on pretrained image foundation models.”
From Correction-aware interactive 3D tumor segmentation with sparse and revisable prompts (2026) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953 · see citing works
Segment anything in medical images
Ma, J., He, Y., Li, F., Han, L., You, C., & Wang, B. (2024) · Nature Communications · doi:10.1038/s41467-024-44824-z · Cited by 2,913 · see citing works
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works
3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation
Gong, S., Zhong, Y., Ma, W., Li, J., Wang, Z., Zhang, J., Heng, P.-A., & Dou, Q. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103324 · Cited by 156 · see citing works
Flagged first, then by citations
Research works printed but not linked: 14
- [2]Zou Xueyan, Yang Jianwei, Zhang Hao, Li Feng, Lin-jie Li, Gao Jianfeng, and Lee Yong Jae, “ Segment everything everywhere all at once,” arXiv preprint arXiv:2304.06718, 2023. · arXiv
- [3]Matsoukas Christos, Haslum Johan Fredin, Moein Sorkhei, Söderberg Magnus, and Smith Kevin, “ What makes transfer learning work for medical images: Feature reuse & other factors,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9225– 9234.
- [6]Pan Junting, Lin Ziyi, Zhu Xiatian, Shao Jing, and Li Hongsheng, “ St-adapter: Parameter-efficient image-to-video transfer learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 26462– 26477, 2022.
- [7]Chen Shoufa, Ge Chongjian, Tong Zhan, Wang Jiangliu, Song Yibing, Wang Jue, and Luo Ping, “ Adaptformer: Adapting vision transformers for scalable visual recognition,” Advances in Neural Information Processing Systems, vol. 35, pp. 16664– 16678, 2022.
- [9]Yang Taojiannan, Zhu Yi, Xie Yusheng, Zhang Aston, Chen Chen, and Li Mu, “ Aim: Adapting image models for efficient video action recognition,” arXiv preprint arXiv:2302.03024, 2023. · arXiv
- [10]Wu Junde, Fu Rao, Fang Huihui, Liu Yuanpei, Wang Zhaowei, Xu Yanwu, Jin Yueming, and Arbel Tal, “ Medical sam adapter: Adapting segment anything model for medical image segmentation,” arXiv preprint arXiv:2304.12620, 2023. 40121809 · arXiv
- [12]Zhang Kaidong and Liu Dong, “ Customized segment anything model for medical image segmentation,” arXiv preprint arXiv:2304.13785, 2023. · arXiv
- [13]Li Hao, Hu Dewei, Liu Han, Wang Jiacheng, and Oguz Ipek, “ Cats Complementary cnn and transformer encoders for segmentation,” in 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI). IEEE, 2022, pp. 1– 5.
- [14]Liu Han, Hu Dewei, Li Hao, and Oguz Ipek, “ Medical image segmentation using deep learning,” in Machine Learning for Brain Disorders, pp. 391– 434. Springer, 2023.
- [15]Kervadec Hoel, Bouchtiba Jihene, Desrosiers Christian, Granger Eric, Dolz Jose, and Ayed Ismail Ben, “ Boundary loss for highly unbalanced segmentation,” in International conference on medical imaging with deep learning. PMLR, 2019, pp. 285– 296.
- [16]Isensee Fabian, Jaeger Paul F, Kohl Simon AA, Petersen Jens, and Maier-Hein Klaus H, “ nnu-net: a self-configuring method for deep learning-based biomedical image segmentation,” Nature methods, vol. 18, no. 2, pp. 203– 211, 2021. 33288961
- [17]Lee Ho Hin, Bao Shunxing, Huo Yuankai, and Landman Bennett A., “ 3d UX-net: A large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation,” in The Eleventh International Conference on Learning Representations, 2023. PMC13597054 42775365
- [18]Zhou Hong-Yu, Guo Jiansen, Zhang Yinghao, Han Xiaoguang, Yu Lequan, Wang Liansheng, and Yu Yizhou, “ nnformer: volumetric medical image segmentation via a 3d transformer,” IEEE Transactions on Image Processing, 2023. 37440404
- [19]Tang Yucheng, Yang Dong, Li Wenqi, Roth Holger R, Landman Bennett, Xu Daguang, Nath Vishwesh, and Hatamizadeh Ali, “ Self-supervised pre-training of swin transformers for 3d medical image analysis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 20730– 20740.
Cited by
Sorted by citations
PRISM: A Promptable and Robust Interactive Segmentation Model with Visual Prompts
Li, H., Liu, H., Hu, D., Wang, J., & Oguz, I. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72384-1_37 · Cited by 12 · see citing works
Self-citationCorrection-aware interactive 3D tumor segmentation with sparse and revisable prompts
Li, H., & Li, H. (2026) · The Visual Computer · doi:10.1007/s00371-026-04560-5
Self-citation
- Authors
- Li, H., Liu, H., Hu, D., Wang, J., & Oguz, I.
- Affiliations
- Vanderbilt University (US)
- Vanderbilt University
- Venue
- 2024 IEEE International Symposium on Biomedical Imaging (ISBI)
- Publisher
- IEEE
- Published
- 2024-05-27
- Volume
- 2024
- Pages
- 1-5
- PMC
- PMC12128788
- OpenAlex
- W4401749938
- Cited by
- 29
- References
- 4
- Citations by year
- 2026: 6 · 2025: 14 · 2024: 9
- FWCI
- 5.99
- Type
- conference paper
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- IEEE International Symposium on Biomedical Imaging (ISBI) (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, other-oa)
- Europe PMC (PubMed Central) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2220401)
- National Institutes of Health
- National Institute of Neurological Disorders and Stroke (U01NS106845)
- Eunice Kennedy Shriver National Institute of Child Health and Human Development (R01HD109739)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- 3D medical image segmentation, Segment Anything Model, vision transformer, lightweight adapters, boundary-aware loss, promise
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
