Retrieval-Augmented Few-Shot Medical Image Segmentation With Foundation Models
Open accessAbstract
Medical image segmentation is crucial for clinical decision-making, but the scarcity of annotated data presents significant challenges. Few-shot segmentation (FSS) methods show promise but often require training on the target domain and struggle to generalize across different modalities. Similarly, adapting foundation models such as the segment anything model (SAM) for medical imaging has limitations, including the need for fine-tuning and domain-specific adaptation. To address these issues, we propose a novel method that adapts DINOv2 and SAM 2 for retrieval-augmented few-shot medical image segmentation. Our approach uses DINOv2's feature as query to retrieve similar samples from limited annotated data, which are then encoded as memories and stored in memory bank. With the memory attention mechanism of SAM 2, the model leverages these memories as conditions to generate accurate segmentation of the target image. We evaluated our framework on three medical image segmentation tasks, demonstrating superior performance and generalizability across various modalities without the need for any retraining or fine-tuning. Overall, this method offers a practical and effective solution for few-shot medical image segmentation and holds significant potential as a valuable annotation tool in clinical applications.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A similar retrieval-augmented approach for few-shot medical image segmentation has been explored in [56], demonstrating its effectiveness in similar scenarios.
“In fact, a similar retrievalaugmented approach for few-shot medical image segmentation has been explored in [56], demonstrating its effectiveness in similar scenarios.”
From ConfIC-RCA: Statistically Grounded Efficient Estimation of Segmentation Quality (2025) · cites this paper as [56]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953 · see citing works
nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021) · Nature Methods · doi:10.1038/s41592-020-01008-z · Cited by 10,477 · see citing works
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 1× · see citing works
“Large-scale language, vision, and multimodal foundation models such as GPTs [24], [25], [26], DINO (self-Distillation with No labels) [21], [27], contrastive language-image pre-training (CLIP) [28], and large language and vision assistant (LLaVA) [29] have dramatically transformed the artificial…”
Segment anything in medical images
Ma, J., He, Y., Li, F., Han, L., You, C., & Wang, B. (2024) · Nature Communications · doi:10.1038/s41467-024-44824-z · Cited by 2,913 · see citing works
Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., Yang, X., Heng, P.-A., Cetin, I., Lekadir, K., Camara, O., Gonzalez Ballester, M. A., et al. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2837502 · Cited by 2,431 · see citing works
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428 · see citing works
Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images
Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H. R., & Xu, D. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-08999-2_22 · Cited by 2,273 · see citing works
U-Net and Its Variants for Medical Image Segmentation: A Review of Theory and Applications
Siddique, N. A., Paheding, S., Elkin, C. P., & Devabhaktuni, V. (2021) · IEEE Access · doi:10.1109/access.2021.3086020 · Cited by 2,060 · see citing works
PANet: Few-Shot Image Semantic Segmentation With Prototype Alignment
Wang, K., Liew, J. H., Zou, Y., Zhou, D., & Feng, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00929 · Cited by 1,581 · see citing works
Medical SAM Adapter: Adapting Segment Anything Model for Medical Image SegmentationPreprint
Wu, J.-D., Fu, R., Fang, H., et al. (2023) · Medical Image Analysis · doi:10.48550/arxiv.2304.12620 · Cited by 1,091 · see citing works
DINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078 · see citing works
Medical Image Segmentation Review: The Success of U-Net
Azad, R., Aghdam, E. K., Rauland, A., Jia, Y., Avval, A. H., Bozorgpour, A., Karimijafarbigloo, S., Cohen, J., Adeli, E., & Merhof, D. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3435571 · Cited by 1,069 · see citing works
Medical SAM adapter: Adapting segment anything model for medical image segmentation
Wu, J., Wang, Z., Hong, M., Ji, W., Fu, H., Xu, Y., Xu, M., & Jin, Y. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103547 · Cited by 909 · see citing works
CHAOS Challenge - combined (CT-MR) healthy abdominal organ segmentation
Kavur, A. E., Gezer, N. S., BARIŞ, M. M., Aslan, S., Conze, P.-H., Groza, V., Pham, D. D., Chatterjee, S., Ernst, P., Özkan, S., et al. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101950 · Cited by 828 · see citing works
Segment anything model for medical image analysis: An experimental study
Mazurowski, M. A., Dong, H., Gu, H., Yang, J., Konz, N., & Zhang, Y. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102918 · Cited by 802 · see citing works
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505 · see citing works
Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.-E., Lomeli, M., Hosseini, L., & Jégou, H. (2026) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3618474 · Cited by 478 · see citing works
SAM 2: Segment Anything in Images and Videos
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al. (2024) · arXiv · doi:10.48550/arxiv.2408.00714 · Cited by 300 · see citing works
Self-supervision with Superpixels: Training Few-Shot Medical Image Segmentation Without Annotation
Ouyang, C., Biffi, C., Chen, C., Kart, T., Qiu, H., & Rueckert, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58526-6_45 · Cited by 285 · see citing works
Hansen, S., Gautam, S., Jenssen, R., & Kampffmeyer, M. C. (2022) · Medical Image Anal. · doi:10.1016/j.media.2022.102385 · Cited by 185 · see citing works
From local to global: a graph rag approach to query-focused summarization
Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., Metropolitansky, D., Ness, R. O., & Larson, J. (2024) · arXiv · doi:10.48550/arxiv.2404.16130 · Cited by 159 · see citing works
Overview and Fundamentals of Medical Image Segmentation
Rogowska, J. (2000) · Elsevier eBooks · doi:10.1016/b978-012077790-7/50009-6 · Cited by 145 · see citing works
Medical Image Segmentation With Limited Supervision: A Review of Deep Network Models
Peng, J., & Wang, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3062380 · Cited by 105 · see citing works
OmniGlue: Generalizable Feature Matching with Foundation Model Guidance
Jiang, H., Karpur, A., Cao, B., Qi-xing Huang, A., & Araújo, A. D. M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01878 · Cited by 90 · see citing works
MedLSAM: Localize and segment anything model for 3D CT images
Lei, W., Xu, W., Li, K., Zhang, X., & Zhang, S. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103370 · Cited by 90 · see citing works
Retrieval Augmented Classification for Long-Tail Visual Recognition
Long, A., Yin, W., Ajanthan, T., Nguyen, D.-V., Purkait, P., Garg, R., Blair, A., Shen, C., & van den Hengel, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00683 · Cited by 86 · see citing works
Q-Net: Query-Informed Few-Shot Medical Image Segmentation
Shen, Q., Li, Y., Jin, J., & Liu, B. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-47724-9_40 · Cited by 64 · see citing works
Few Shot Medical Image Segmentation with Cross Attention Transformer
Lin, Y., Chen, Y., Cheng, K.-T. T., & Chen, H. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43895-0_22 · Cited by 60 · see citing works
A Unified Framework for Generalized Low-Shot Medical Image Segmentation With Scarce Data
Cui, H., Wei, D., Ma, K., Gu, S., & Zheng, Y. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3045775 · Cited by 54 · see citing works
Deep learning-based medical image segmentation with limited labels
Chi, W., Ma, L., Wu, J., Chen, M., Lu, W., & Gu, X. (2020) · Physics in Medicine and Biology · doi:10.1088/1361-6560/abc363 · Cited by 50 · see citing works
Learning Customized Visual Models with Retrieval-Augmented Knowledge
Liu, H., Son, K., Yang, J., Liu, C., Gao, J., Lee, Y. J., & Li, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01454 · Cited by 41 · see citing works
DINO-Tracker: Taming DINO for Self-supervised Point Tracking in a Single Video
Tumanyan, N., Singer, A., Bagon, S., & Dekel, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73347-5_21 · Cited by 38 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
MedLSAM: Localize and Segment Anything Model for 3D CT Images
Lei, W., Wei, X., Zhang, X., Li, K., & Zhang, S. (2023) · arXiv · doi:10.48550/arxiv.2306.14752
SAM3D: Segment Anything Model in Volumetric Medical Images
Bui, N.-T., Hoang, D.-H., Tran, M.-T., Doretto, G., Adjeroh, D., Patel, B., Choudhary, A., & Le, N. (2023) · arXiv · doi:10.48550/arxiv.2309.03493
DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video
Tumanyan, N., Singer, A., Bagon, S., & Dekel, T. (2024) · arXiv · doi:10.48550/arxiv.2403.14548
Retrieval-Augmented Diffusion Models
Blattmann, A., Rombach, R., Oktay, K., Müller, J., & Ommer, B. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1114
Flagged first, then by citations
Research works printed but not linked: 3
- P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al., "Retrieval- augmented generation for knowledge-intensive nlp tasks," Advances in Neural Information Processing Systems, vol. 33, pp. 9459-9474, 2020.
- A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., "Learning transferable visual models from natural language supervision," in International conference on machine learning. PMLR, 2021, pp. 8748-8763.
- C. Ryali, Y.-T. Hu, D. Bolya, C. Wei, H. Fan, P.-Y. Huang, V. Aggarwal, A. Chowdhury, O. Poursaeed, J. Hoffman et al., "Hiera: A hierarchi- cal vision transformer without the bells-and-whistles," in International Conference on Machine Learning. PMLR, 2023, pp. 29 441-29 454.
Cited by
Sorted by citations
Draft-Ing Architectural Design Decisions Using LlmsPreprint
Dhar, R., Kakran, A., Karan, A., Vaidhyanathan, K., & Varma, V. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5217607 · Cited by 11 · see citing works
Comprehensive review of segment anything model across multiple domains
Wang, X., Duan, T., Ouyang, G., Hao, W., Mu, L., & Zhang, X. (2025) · Digital Signal Processing · doi:10.1016/j.dsp.2025.105459 · Cited by 2 · see citing works
Wu, S., Liu, R., Wei, H., Liu, L., Wen, J., & Miao, Q. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3649537 · Cited by 1 · see citing works
Learning gated experts for segment anything in the wild
Guo, Y., Guo, H., Dai, T., Wang, Z., Chen, B., & Xia, S. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113064 · Cited by 1 · see citing works
Guest Editorial Special Issue on Advancements in Foundation Models for Medical Imaging
Liu, T., Shen, D. G., Ye, J. C., de Bruijne, M., & Liu, W. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3613074 · Cited by 1 · see citing works
ConfIC-RCA: Statistically Grounded Efficient Estimation of Segmentation Quality
Cosarinsky, M., Billot, R., Mansilla, L., Jimenez, G., Gaggión, N., Fu, G., Tirer, T., & Ferrante, E. (2025) · arXiv
“To implement this, we embed all the images from the reference database using DINOv2 embeddings [54] and then create a vector database using FAISS [55], which enables e ffi cient similarity search.”
Multi-level hierarchical prototype for few-shot medical image segmentation
Sun, S., Yang, J., Tang, L.-Z., Feng, Y., & Qi, Y. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.108500
Self-citationRAU: Reference-Based Anatomical Understanding with Vision Language Models
Li, Y., Liu, Y., Guo, J., Zhao, L., Zhang, Z., Chen, X., Mailhé, B., Mukherjee, A., Chen, T., & Sun, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37041-9_19
Self-citationToward Content-Based Indexing and Retrieval of Head and Neck CT With Abscess Segmentation
Dao, T. T. P., Nguyen, T.-C., Do, T.-L., Viet, T. H., Thành, N. C., Thuan, H. N., Nguyen, D. V. C., Pham, M.-K., Tran, M.-K., Huynh, V.-T., et al. (2025) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1109/cbmi66578.2025.11339277
Nguyen, L.-P., Nguyen, T.-D., & Nguyen, V.-T. (2025) · RIVF International Conference on Computing and Communication Technologies (RIVF) · doi:10.1109/rivf68649.2025.11365102
Enhancing Histology Image Segmentation Using Improved Retrieval Augmented Generation Technique
Hazra, A., Bajpayee, A., & Das, N. (2025) · IEEE International Symposium on Sustainable Energy, Signal Processing and Cyber Security (iSSSC) · doi:10.1109/isssc66652.2025.11388798
FUSE-RAG: Few-Shot Universal Segmentation with Retrieval-Augmented Generation for Medical Imaging
Allaoui, M. L., Allili, M. S., & Belaid, A. (2026) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3807452
Not Quite Anything: Overcoming SAM’s Limitations for 3D Medical Imaging
Moore, K. (2026) · EPiC series in technology · doi:10.29007/zckn
TumorAL: Evidence-aware active learning for 3D tumor segmentation
Wang, H., Leng, J., Li, S., Zhao, Y., Li, W., Li, W., & Gao, X. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134239
Wu, Y., Zhang, J., Liu, S., Luo, X., Lyu, G., & Shen, L. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133421
Žídek, J., Lipový, B., Pavliňáková, V., Knoz, M., Holoubek, J., Izsák, D., Váňa, V., Herudek, J., Jeklová, E., Koutná, I., et al. (2026) · Research Square · doi:10.21203/rs.3.rs-10193916/v1
Sun, Z., Ye, L., Xu, Y., & Li, Y. (2026) · Frontiers in Neurology · doi:10.3389/fneur.2026.1895626
Retrieval-Guided Semi-supervised Federated Learning for Medical Image Segmentation
Park, H., Kim, S., Arsalane, W., An, S., & Park, S. H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38069-2_50
Xia, Q., Sun, X., Jiang, B., Yun, Z., Chen, D., & Zhai, S. (2026) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2026.102835
OneRef ‐Seg: A Single‐Reference Framework for Efficient Medical Image Segmentation
Tang, L., Yang, W., & Li, J. (2026) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.70466
- Authors
- Zhao, L., Chen, X., Chen, E. Z., Liu, Y., Chen, T., & Sun, S.
- Affiliations
- United Imaging Intelligence, Boston, MA, USA
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2025-10
- Volume
- 36
- Issue
- 10
- Pages
- 17693-17701
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W4410986053
- Cited by
- 19
- References
- 26
- Citations by year
- 2026: 12 · 2025: 7
- FWCI
- 7.69
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- retrieval-augmented learning, foundation models, Segment Anything Model, DINOv2, memory bank, cross-modal generalization, annotation tools, limited labeled data
- MeSH terms
- Algorithms, Diagnostic Imaging, Humans, Image Processing, Computer-Assisted, Databases, Factual, Information Storage and Retrieval, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
