CoChat
Sign Up
PreprintCited by 3Multimodal Machine Learning ApplicationsMedical Image Segmentation TechniquesArtificial Intelligence in Healthcare and Education

MediSee: Reasoning-based Pixel-level Perception in Medical Images

Open access
Tong, Qinyue, Lu, Ziqian, Liu, Jun, Zheng, Yangming, Lu, Zheming·2025·arXiv
DOI 10.48550/arxiv.2504.11008arXiv 2504.11008
Open this paper in CoChat — free accountOpen PDF

Abstract

Despite remarkable advancements in pixel-level medical image perception, existing methods are either limited to specific tasks or heavily rely on accurate bounding boxes or text labels as input prompts. However, the medical knowledge required for input is a huge obstacle for general public, which greatly reduces the universality of these methods. Compared with these domain-specialized auxiliary information, general users tend to rely on oral queries that require logical reasoning. In this paper, we introduce a novel medical vision task: Medical Reasoning Segmentation and Detection (MedSD), which aims to comprehend implicit queries about medical images and generate the corresponding segmentation mask and bounding box for the target object. To accomplish this task, we first introduce a Multi-perspective, Logic-driven Medical Reasoning Segmentation and Detection (MLMR-SD) dataset, which encompasses a substantial collection of medical entity targets along with their corresponding reasoning. Furthermore, we propose MediSee, an effective baseline model designed for medical reasoning segmentation and detection. The experimental results indicate that the proposed method can effectively address MedSD with implicit colloquial queries and outperform traditional medical referring segmentation methods.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2504.11008

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Dependence on joint fine-tuning of the MLLM and external pixel decoders heightens the risk of catastrophic forgetting, resulting in strong in-domain performance but weak cross-domain generalization.

    “Their dependence on joint fine-tuning of the MLLM and external pixel decoders heightens the risk of catastrophic forgetting , resulting in strong in-domain performance but weak cross-domain generalization.”

    From IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (2026) · cites this paper as [38]

    Section 4.3 (page unavailable) describes fine-tuning the MLLM with LoRA while fully fine-tuning the mask decoder and training the detection decoder. It does not say that this creates a risk of catastrophic forgetting or produces strong in-domain but weak cross-domain performance; searches for those outcomes found no supporting passage. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Existing medical MLLM approaches introduce implicit segmentation tokens and require joint fine-tuning of the MLLM and external pixel decoders, increasing the risk of catastrophic forgetting and limiting out-of-domain generalization.

    “First, they introduce implicit segmentation tokens and require joint fine-tuning of the MLLM and external pixel decoders, increasing the risk of catastrophic forgetting and limiting out-of-domain generalization.”

    From IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (2026) · cites this paper as [38]

    Section 4.2 says existing MLLM-based methods generate a special segmentation token whose hidden representation is used by a downstream decoder to produce a mask. Section 4.3 describes fine-tuning MediSee’s own language model and mask decoder, but does not establish that existing approaches require joint fine-tuning or that this causes catastrophic forgetting or limits out-of-domain generalization. Page numbers are unavailable in our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 52

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). · arXiv
  2. Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ron- neberger, Ronald M Summers, et al. 2022. The medical segmentation decathlon. Nature communications 13, 1 (2022), 4128.
  3. Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Kalpathy-Cramer, Felipe C Kitamura, Sarthak Pati, et al. 2021. The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021). · arXiv
  4. Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025. Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling. arXiv preprint arXiv:2501.17811 (2025). · arXiv
  5. Junlong Cheng, Bin Fu, Jin Ye, Guoan Wang, Tianbin Li, Haoyu Wang, Ruoyu Li, He Yao, Junren Chen, JingWen Li, et al. 2024. Interactive medical image segmentation: A benchmark dataset and baseline. arXiv preprint arXiv:2411.12814 (2024). · arXiv
  6. Junlong Cheng, Jin Ye, Zhongying Deng, Jianpin Chen, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, et al. 2023. Sam-med2d. arXiv preprint arXiv:2308.16184 (2023). · arXiv
  7. Michal Drozdzal, Eugene Vorontsov, Gabriel Chartrand, Samuel Kadoury, and Chris Pal. 2016. The importance of skip connections in biomedical image seg- mentation. In International Workshop on Deep Learning in Medical Image Analysis. Springer, 179-187.
  8. Yuxin Du, Fan Bai, Tiejun Huang, and Bo Zhao. 2024. Segvol: Universal and inter- active volumetric medical image segmentation. Advances in Neural Information Processing Systems 37 (2024), 110746-110783.
  9. Nicholas Heller, Fabian Isensee, Dasha Trofimova, Resha Tejpaul, Zhongchen Zhao, Huai Chen, Lisheng Wang, Alex Golts, Daniel Khapun, Daniel Shats, et al. 2023. The kits21 challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase ct. arXiv preprint arXiv:2307.01984 (2023). · arXiv
  10. Murtadha D Hssayeni, Muayad S Croock, Aymen D Salman, Hassan Falah Al- Khafaji, Zakaria A Yahya, and Behnaz Ghoraani. 2020. Intracranial hemorrhage segmentation using a deep convolutional model. Data 5, 1 (2020), 14.
  11. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR 1, 2 (2022), 3.
  12. Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier- Hein. 2021. nnU-Net: a self-configuring method for deep learning-based biomed- ical image segmentation. Nature methods 18, 2 (2021), 203-211.
  13. David A Jaffray, Felicia Knaul, Michael Baumann, and Mary Gospodarowicz. 2023. Harnessing progress in radiotherapy for global cancer control. Nature Cancer 4, 9 (2023), 1228-1238.
  14. Donggon Jang, Yucheol Cho, Suin Lee, Taehyeon Kim, and Dae-Shik Kim. 2025. MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation. arXiv preprint arXiv:2503.13881 (2025). · arXiv
  15. Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision. 4015-4026.
  16. Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. 2024. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9579- 9589.
  17. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023. Llava-med: Train- ing a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36 (2023), 28541-28564.
  18. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual in- struction tuning. Advances in neural information processing systems 36 (2023), 34892-34916.
  19. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. 2024. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision. Springer, 38-55.
  20. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017). · arXiv
  21. Timo Lüddecke and Alexander Ecker. 2022. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 7086-7096.
  22. Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. 2024. Segment anything in medical images. Nature Communications 15, 1 (2024), 654.
  23. Jun Ma, Yao Zhang, Song Gu, Cheng Ge, Shihao Mae, Adamo Young, Cheng Zhu, Xin Yang, Kangkang Meng, Ziyan Huang, et al. 2024. Unleashing the strengths of unlabelled data in deep learning-assisted pan-cancer abdominal organ quantification: the FLARE22 challenge. The Lancet Digital Health 6, 11 (2024), e815-e826.
  24. Victor Nauffal, Marcus DR Klarqvist, Matthew C Hill, Danielle F Pace, Paolo Di Achille, Seung Hoan Choi, Joel T Rämö, James P Pirruccello, Pulkit Singh, Shinwan Kany, et al. 2024. Noninvasive assessment of organ-specific and shared pathways in multi-organ fibrosis using T1 mapping. Nature Medicine 30, 6 (2024), 1749-1760.
  25. Alireza Norouzi, Mohd Shafry Mohd Rahim, Ayman Altameem, Tanzila Saba, Abdolvahab Ehsani Rad, Amjad Rehman, and Mueen Uddin. 2014. Medical image segmentation methods, algorithms, and applications. IETE Technical Review 31, 3 (2014), 199-213.
  26. Renjie Pi, Lewei Yao, Jiahui Gao, Jipeng Zhang, and Tong Zhang. 2024. Per- ceptiongpt: Effectively fusing visual perception into llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 27124-27133.
  27. Rui Qian, Xin Yin, and Dejing Dou. 2024. Reasoning to Attend: Try to Understand How< SEG> Token Works. arXiv preprint arXiv:2412.17741 (2024). · arXiv
  28. Martin Rajchl, Matthew CH Lee, Ozan Oktay, Konstantinos Kamnitsas, Jonathan Passerat-Palmbach, Wenjia Bai, Mellisa Damodaram, Mary A Rutherford, Joseph V Hajnal, Bernhard Kainz, et al. 2016. Deepcut: Object segmentation from bounding box annotations using convolutional neural networks. IEEE transactions on medical imaging 36, 2 (2016), 674-683.
  29. KKD Ramesh, G Kiran Kumar, K Swapna, Debabrata Datta, and S Suman Ra- jest. 2021. A review of medical image segmentation algorithms. EAI Endorsed Transactions on Pervasive Health & Technology 7, 27 (2021).
  30. Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. 2024. Glamm: Pixel grounding large multimodal model. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13009-13018.
  31. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deep- speed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 3505-3506.
  32. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolu- tional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18. Springer, 234-241.
  33. Rina D Rudyanto, Sjoerd Kerkstra, Eva M Van Rikxoort, Catalin Fetita, Pierre-Yves Brillet, Christophe Lefevre, Wenzhe Xue, Xiangjun Zhu, Jianming Liang, Ilkay Öksüz, et al. 2014. Comparing algorithms for automated vessel segmentation in computed tomography scans of the lung: the VESSEL12 study. Medical image analysis 18, 7 (2014), 1217-1232.
  34. Anindo Saha, Matin Hosseinzadeh, and Henkjan Huisman. 2021. End-to-end prostate cancer detection in bpMRI via 3D CNNs: effects of attention mechanisms, clinical priori and decoupled false positive reduction. Medical image analysis 73 (2021), 102155.
  35. T Sakinis, F Milletari, H Roth, P Korfiatis, P Kostandy, K Philbrick, Z Akkus, Z Xu, D Xu, and BJ Erickson. 1903. Interactive segmentation of medical images through fully convolutional neural networks. arXiv 2019. arXiv preprint arXiv:1903.08205 (1903). · arXiv
  36. Arnaud Arindra Adiyoso Setio, Alberto Traverso, Thomas De Bel, Moira SN Berens, Cas Van Den Bogaard, Piergiorgio Cerello, Hao Chen, Qi Dou, Maria Evelina Fantacci, Bram Geurts, et al. 2017. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge. Medical image analysis 42 (2017), 1-13.
  37. Guotai Wang, Maria A Zuluaga, Rosalind Pratt, Michael Aertsen, Tom Doel, Maria Klusmann, Anna L David, Jan Deprest, Tom Vercauteren, and Sébastien Ourselin. 2016. Slic-Seg: A minimally interactive segmentation of the placenta from sparse and motion-corrupted fetal MRI in multiple views. Medical image analysis 34 (2016), 137-147.
  38. Junchi Wang and Lei Ke. 2024. Llm-seg: Bridging image segmentation and large language model reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1765-1774.
  39. Risheng Wang, Tao Lei, Ruixia Cui, Bingtao Zhang, Hongying Meng, and Asoke K Nandi. 2022. Medical image segmentation using deep learning: A survey. IET image processing 16, 5 (2022), 1243-1267.
  40. XuDong Wang, Shaolun Zhang, Shufan Li, Konstantinos Kallidromitis, Kehan Li, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. 2024. SegLLM: Multi-round Reasoning Segmentation. arXiv preprint arXiv:2410.18923 (2024). · arXiv
  41. Yueyue Wang, Liang Zhao, Manning Wang, and Zhijian Song. 2019. Organ at risk segmentation in head and neck CT images using a two-stage segmentation framework based on 3D U-Net. IEEE Access 7 (2019), 144591-144602.
  42. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reason- ing in large language models. Advances in neural information processing systems 35 (2022), 24824-24837.
  43. Hallee E Wong, Marianne Rakic, John Guttag, and Adrian V Dalca. 2024. Scrib- bleprompt: fast and flexible interactive segmentation for any biomedical image. In European Conference on Computer Vision. Springer, 207-229.
  44. Ke Yan, Xiaosong Wang, Le Lu, and Ronald M Summers. 2018. DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning. Journal of medical imaging 5, 3 (2018), 036501-036501.
  45. Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. 2023. LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Model. arXiv preprint arXiv:2312.17240 (2023). · arXiv
  46. Jin Ye, Junlong Cheng, Jianpin Chen, Zhongying Deng, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, et al. 2023. Sa-med2d-20m dataset: Segment anything in 2d medical imaging with 20 million masks. arXiv preprint arXiv:2311.11969 (2023). · arXiv
  47. Ao Zhang, Yuan Yao, Wei Ji, Zhiyuan Liu, and Tat-Seng Chua. 2023. Next-chat: An lmm for chat, detection and segmentation. arXiv preprint arXiv:2311.04498 (2023). · arXiv
  48. Jian Zhang, Yinghuan Shi, Jinquan Sun, Lei Wang, Luping Zhou, Yang Gao, and Dinggang Shen. 2021. Interactive medical image segmentation via a point-based interaction. Artificial Intelligence in Medicine 111 (2021), 101998.
  49. Ziheng Zhao, Yao Zhang, Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. One model to rule them all: Towards universal segmentation for medical images with text prompts. arXiv preprint arXiv:2312.17183 (2023). · arXiv
  50. Tianfei Zhou, Liulei Li, Gustav Bredell, Jianwu Li, Jan Unkelbach, and Ender Konukoglu. 2023. Volumetric memory network for interactive medical image segmentation. Medical Image Analysis 83 (2023), 102599.
  51. Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. 2018. Unet++: A nested u-net architecture for medical image segmentation. In Deep learning in medical image analysis and multimodal learning for clinical decision support: 4th international workshop, DLMIA 2018, and 8th international workshop, ML-CDS 2018, held in conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, proceedings 4. Springer, 3-11.
  52. Xun Zhu, Ying Hu, Fanbin Mo, Miao Li, and Ji Wu. 2024. Uni-med: a unified medical generalist foundation model for multi-task learning via connector-MoE. arXiv preprint arXiv:2409.17508 (2024). · arXiv

Cited by

Sorted by citations

Authors
Tong, Q., Lu, Z., Liu, J., Zheng, Y., & Lu, Z.
Affiliations
  • Zhejiang University (CN)
  • Zhejiang Sci-Tech University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Pages
2742-2751
DOI
10.48550/arxiv.2504.11008
arXiv
2504.11008
OpenAlex
W4415541250
Cited by
3
References
0
Citations by year
2026: 1 · 2025: 2
FWCI
2.26
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • ACM International Conference on Multimedia (ACM MM) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
medical image segmentation, medical object detection, natural language queries, referring segmentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review