CoChat
Sign Up
PreprintCited by 6Multimodal Machine Learning ApplicationsArtificial Intelligence in Healthcare and EducationMedical Image Segmentation Techniques

UniBiomed: A Universal Foundation Model for Grounded Biomedical Image Interpretation

Open access
Wu, Linshan, Nie, Yuxiang, He, Sunan, Zhuang, Jiaxin, Luo, Luyang, Li, Tao, Xie, Zhuoyao, Chen, Dexuan +7 more·2025·arXiv
DOI 10.48550/arxiv.2504.21336arXiv 2504.21336PMC PMC13396234
Open this paper in CoChat — free account

Abstract

The integration of AI-assisted biomedical image analysis into clinical practice demands AI-generated findings that are not only accurate but also interpretable to clinicians. However, existing biomedical AI models generally lack the ability to simultaneously generate diagnostic findings and localize corresponding biomedical objects. This limitation makes it challenging for clinicians to correlate AI-generated findings with visual evidence (e.g., tiny lesions) in images and interpret the results of AI models. To address this challenge, we introduce UniBiomed, the first universal foundation model for grounded biomedical image interpretation, which is capable of generating accurate diagnostic findings and simultaneously segmenting the corresponding biomedical targets. UniBiomed is based on a novel integration of Multi-modal Large Language Model and Segment Anything Model, which can effectively unify diverse biomedical tasks in universal training for advancing grounded interpretation. To develop UniBiomed, we curate a large-scale dataset comprising over 27 million triplets of images, region annotations, and text descriptions across ten biomedical imaging modalities. Extensive validation on 70 internal and 14 external datasets demonstrated the state-of-the-art performance of UniBiomed in diverse biomedical tasks, including image segmentation, disease recognition, region-aware diagnosis, vision question answering, and report generation. In summary, UniBiomed is a powerful and versatile biomedical foundation model, unlocking the untapped grounded interpretation capability for optimizing AI-assisted biomedical image analysis.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2504.21336

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • UniBiomed produces incorrect textual reasoning and responses, describing objects that do not exist in the image and generating low-quality segmentation masks.

    “UniBiomed [41] produces incorrect textual reasoning and responses, describing objects that do not exist in the image and generating low-quality segmentation masks.”

    From IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (2026) · cites this paper as [41]

    The Method section describes UniBiomed as generating precise segmentation masks and detailed diagnostic descriptions, rather than the low-quality masks and incorrect responses attributed to it. The Discussion acknowledges limits in rare-disease coverage and clinical validation, but does not report descriptions of nonexistent objects or incorrect textual reasoning. (LLM, conf=0.77)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Existing medical MLLM approaches introduce implicit segmentation tokens and require joint fine-tuning of the MLLM and external pixel decoders, increasing the risk of catastrophic forgetting and limiting out-of-domain generalization.

    “First, they introduce implicit segmentation tokens and require joint fine-tuning of the MLLM and external pixel decoders, increasing the risk of catastrophic forgetting and limiting out-of-domain generalization.”

    From IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (2026) · cites this paper as [41]

    The Methods section says UniBiomed adopts a [SEG] token following previous methods and fine-tunes its LLM and SAM prompt encoder and mask decoder. It does not say that existing medical MLLM approaches generally require this joint fine-tuning or that it increases catastrophic forgetting or limits out-of-domain generalization. The Discussion instead says recent medical MLLMs generally cannot segment the objects they describe. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Dependence on joint fine-tuning of the MLLM and external pixel decoders heightens the risk of catastrophic forgetting, resulting in strong in-domain performance but weak cross-domain generalization.

    “Their dependence on joint fine-tuning of the MLLM and external pixel decoders heightens the risk of catastrophic forgetting , resulting in strong in-domain performance but weak cross-domain generalization.”

    From IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (2026) · cites this paper as [41]

    The Methods section describes training an MLLM together with SAM’s prompt encoder and mask decoder, supporting the architectural part of the claim. It says LoRA and frozen vision encoders are used to preserve pretrained knowledge, but does not report catastrophic forgetting or a resulting pattern of strong in-domain and weak cross-domain performance. The Discussion instead reports better performance across diverse biomedical tasks; page numbers are unavailable in our copy. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 70

  1. Luo, L. et al. Deep mining external imperfect data for chest x-ray disease screening. IEEE Trans. Med. Imaging 39, 3583–3594 (2020).
  2. Antonelli, M. et al. The medical segmentation decathlon. Nat. Commun. 13, 4128 (2022).
  3. He, S. et al. Towards generalizable AI in medicine via generalist-specialist collaboration. Nat Biomed. Eng. 1–15 https://doi.org/10.1038/s41551-026-01653-3 (2026). · DOI
  4. He, S. et al. MEDDR: diagnosis-guided bootstrapping for large-scale medical vision-language learning. (CoRR, 2024).
  5. Li, C. et al. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Adv. Neural Inf. Process. Syst. 36, 28541–28564 (2023).
  6. Ma, J. et al. Segment anything in medical images. Nat. Commun. 15, 654 (2024).
  7. Zhao, T. et al. A foundation model for joint segmentation, detection, and recognition of biomedical objects across nine modalities. Nat. Methods (2024). https://doi.org/10.1038/s41592-024-02499-w · DOI
  8. Zhuang, J., Wu, L., Ni, X., Wang, X., Wang, L. & Chen, H. Bio2vol: adapting 2D biomedical foundation models for volumetric medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 24–34 (MICCAI, 2025).
  9. Du, Y., Bai, F., Huang, T. & Zhao, B. Segvol: universal and interactive volumetric medical image segmentation. Adv. Neural Inf. Process. Syst. 37, 110746–110783 (2024).
  10. Zhao, Z. et al. Large-vocabulary segmentation for medical images with text prompts. npj Digit/ Med. 8, 566 (2025).
  11. Liu, H., Li, C., Wu, Q. & Lee, Y. J. Visual instruction tuning. Adv. Neural Inf. Process. Syst. 36, 34892–34916 (2023).
  12. Chen, Z. et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 (2024). · arXiv
  13. Kirillov, A. et al. Segment anything. In Proc. IEEE/CVF International Conference on Computer Vision, pp. 4015–4026 (IEEE, 2023).
  14. Ravi, N. et al. Sam 2: Segment anything in images and videos. (ICLR, 2024).
  15. Xie, Y. et al. Medtrinity-25m: a large-scale multimodal dataset with multigranular annotations for medicine. (ICLR, 2025).
  16. Hamamci, I. E. et al. A foundation model utilizing chest ct volumes and radiology reports for supervised-level zero-shot detection of abnormalities. arXiv preprint arXiv:2403.17834 (2024). · arXiv
  17. Yuan, H. et al. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.04001 (2025). · arXiv
  18. Lai, X. et al. Lisa: Reasoning segmentation via large language model. In: Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579–9589 (IEEE, 2024).
  19. Wang, L. et al. Interpretable bilingual multimodal large language model for diverse biomedical tasks. (ICLR, 2025).
  20. Huang, X. et al. Towards a multimodal large language model with pixel-level insight for biomedicine. In Proc. AAAI Conference on Artificial Intelligence, vol. 39, pp. 3779–3787 (AAAI, 2025).
  21. Radford, A. et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748–8763 (ICML, 2021).
  22. Chen, Z. et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185–24198 (IEEE, 2024).
  23. Bai, J. et al. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 (2023). · arXiv
  24. Li, J., Li, D., Savarese, S. & Hoi, S. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In: International Conference on Machine Learning, pp. 19730–19742 (ICML, 2023).
  25. Touvron, H. et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). · arXiv
  26. Liu, S. et al. Grounding dino: marrying dino with grounded pre-training for open-set object detection. In: European Conference on Computer Vision, pp. 38–55 (ECCV, 2024).
  27. Li, L. H. et al. Grounded language-image pre-training. In: Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965–10975 (IEEE, 2022).
  28. Zou, X. et al. Segment everything everywhere all at once. Adv. Neural Inf. Process. Syst. 36, 19769–19782 (2023).
  29. Zhang, T. et al. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. Adv. Neural Inf. Process. Syst. 37, 71737–71767 (2024).
  30. Rasheed, H. et al. Glamm: Pixel grounding large multimodal model. In: Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13009–13018 (IEEE, 2024).
  31. Bai, F., Du, Y., Huang, T., Meng, M.Q.-H. & Zhao, B. M3d: advancing 3D medical image analysis with multi-modal large language models. arXiv preprint arXiv:2404.00578 (2024). · arXiv
  32. Zhang, X. et al. Radgenome-chest ct: A grounded vision-language dataset for chest CT analysis. arXiv preprint arXiv:2404.16754 (2024). · arXiv
  33. Hamamci, I. E. et al. Developing generalist foundation models from a multimodal dataset for 3d computed tomography. arXiv preprint arXiv:2403.17834 (2024). · arXiv
  34. Yuan, Y. et al. Osprey: Pixel understanding with visual instruction tuning. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28202–28211 (IEEE, 2024).
  35. Cai, M. et al. Vip-llava: Making large multimodal models understand arbitrary visual prompts. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12914–12923 (IEEE, 2024).
  36. Tanida, T., Müller, P., Kaissis, G. & Rueckert, D. Interactive and explainable region-guided radiology report generation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7433–7442 (IEEE, 2023).
  37. Wu, C. et al. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. Nat. Commun. 16, 7866 (2025).
  38. Shui, Z. et al. Large-scale and fine-grained vision-language pre-training for enhanced CT image understanding. In International Conference on Learning Representations (ICML, 2025).
  39. Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318 (ACL, 2002).
  40. Banerjee, S., Lavie, A. Meteor: an automatic metric for mt evaluation with improved correlation with human judgments. In Proc. ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation And/or Summarization, pp. 65–72 (ACL, 2005).
  41. Lin, C.-Y. Rouge: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (ACL, 2004).
  42. Nie, Y. et al. Conceptclip: Towards trustworthy medical AI via concept-enhanced contrastive langauge-image pre-training. arXiv preprint arXiv:2501.15579 (2025). · arXiv
  43. Nie, Y. et al. An explainable biomedical foundation model via large-scale concept-enhanced vision-language pre-training. arXiv preprint arXiv:2501.15579 (2025). · arXiv
  44. Zhuang, J. et al. Mim: mask in mask self-supervised pre-training for 3d medical image analysis. IEEE Transactions on Medical Imaging (IEEE, 2025).
  45. Wu, L. et al. Large-scale generative tumor synthesis in computed tomography images for improving tumor recognition. Nat. Commun. 16, 11053 (2025).
  46. Wu, L., Zhuang, J., Chen, H. Glance and focus reinforcement for pan-cancer screening. In International Conference on Learning Representations (ICLR, 2026).
  47. Ni, X. et al. Mg-3D: multi-grained knowledge-enhanced 3d medical vision-language pre-training. Med. Image. Anal. 111, 104027 (2024).
  48. Zhu, J., Qi, Y., Wu, J. Medical sam 2: segment medical images as video via segment anything model 2. arXiv preprint arXiv:2408.00874 (2024). · arXiv
  49. Fallahpour, A., Ma, J., Munim, A., Lyu, H. & WANG, B. Medrax: medical reasoning agent for chest x-ray. In: Forty-second International Conference on Machine Learning (ICML, 2025).
  50. Li, B. et al. Mmedagent: learning to use medical tools with multi-modal agent. Findings of the Association for Computational Linguistics: EMNLP 2024. 8745–8760 (2024).
  51. Nath, V. et al. Vila-m3: enhancing vision-language models with medical expert knowledge. In Proc. Computer Vision and Pattern Recognition Conference, pp. 14788–14798 (CVPR, 2025).
  52. Dosovitskiy, A. et al. An image is worth 16 × 16 words: transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR, 2020).
  53. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (long and Short Papers), pp. 4171–4186 (ACL, 2019).
  54. Ryali, C. et al. Hiera: A hierarchical vision transformer without the bells-and-whistles. In: International Conference on Machine Learning, pp. 29441–29454 (ICML, 2023).
  55. Hu, E. J. et al. Lora: low-rank adaptation of large language models. Int. Conf. Learn. Representations 1, 3 (2022).
  56. Xu, W. et al. Lingshu: a generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044 (2025). · arXiv
  57. Jiang, S. et al. Hulu-med: a transparent generalist model towards holistic medical vision-language understanding. arXiv preprint arXiv:2510.08668 (2025). · arXiv
  58. Wu, L., Zhuang, J. & Chen, H. Large-scale 3D medical image pre-training with geometric context priors. IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE, 2025).
  59. Wu, L., Zhuang, J. & Chen, H. Voco: a simple-yet-effective volume contrastive learning framework for 3D medical image analysis. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22873–22882 (IEEE, 2024).
  60. Paszke, A. et al. Pytorch: an imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst. 32, 1–12 (2019).
  61. Wang, X. et al. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 2097–2106 (IEEE, 2017).
  62. Silva-Rodríguez, J., Colomer, A., Sales, M. A., Molina, R. & Naranjo, V. Going deeper through the Gleason scoring scale: an automatic end-to-end system for histology prostate grading and cribriform pattern detection. Computer Methods Prog. Biomed.195, 105637 (2020).
  63. Chen, G., Bai, J., Ou, Z., Lu, Y. & Wang, H. PSFHS: intrapartum ultrasound image dataset for AI-based segmentation of pubic symphysis and fetal head. Sci. Data 11, 436 (2024).
  64. Orlando, J. I. et al. Refuge challenge: a unified framework for evaluating automated methods for glaucoma assessment from fundus photographs. Med. Image Anal. 59, 101570 (2020).
  65. Gutman, D. et al. Skin lesion analysis toward melanoma detection: A challenge at the international symposium on biomedical imaging (ISBI) 2016, hosted by the international skin imaging collaboration (ISIC). arXiv preprint arXiv:1605.01397 (2016). · arXiv
  66. Kulyabin, M. et al. OCTDL: optical coherence tomography dataset for image-based deep learning methods. Sci. Data 11, 365 (2024).
  67. Gatidis, S. et al. A whole-body FDG-PET/CT dataset with manually annotated tumor lesions. Sci. Data 9, 601 (2022).
  68. Khaled, R. et al. Categorized digital database for low energy and subtracted contrast enhanced spectral mammography images. The Cancer Imaging Archive 16 (2021). https://doi.org/10.7937/29kw-ae92 · DOI
  69. Roth, H. R. et al. Rapid artificial intelligence solutions in a pandemic-? The COVID-19-20 lung CT lesion segmentation challenge. Med. Image Anal. 82, 102605 (2022).
  70. Han, C. et al. Multi-layer pseudo-supervision for histopathology tissue semantic segmentation using patch-level classification labels. Med. Image Anal. 80, 102487 (2022).

Cited by

Sorted by citations

Authors
Wu, L., Nie, Y., He, S., Zhuang, J., Luo, L., Li, T., Xie, Z., Chen, D., Zhao, Y., Mahboobani, N., Vardhanabhuti, V., Chan, R. C. K., Peng, Y., Rajpurkar, P., & Chen, H.
Affiliations
  • Hong Kong University of Science and Technology (HK)
  • Harvard University (US)
  • Third Affiliated Hospital of Southern Medical University (CN)
  • Chinese University of Hong Kong (HK)
  • University of Hong Kong (HK)
  • Weill Cornell Medicine (US)
Corresponding author
Chen, Hao
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Volume
17
Issue
1
ISSN
2041-1723 (ISSN-L)
DOI
10.48550/arxiv.2504.21336
arXiv
2504.21336
PMC
PMC13396234
OpenAlex
W7163562394
Cited by
6
References
0
Citations by year
2026: 6
FWCI
12.90
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Innovation and Technology Commission (ITC PD/17-9)
  • Hong Kong University of Science and Technology (ITCPD/17-9)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
Keywords
multimodal large language models, Segment Anything Model, image segmentation, region localization, multimodal dataset, explainable AI, clinical AI
MeSH terms
Large Language Models, Artificial Intelligence, Humans, Image Interpretation, Computer-Assisted, Databases, Factual

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review