CoChat
Sign Up
PreprintCited by 1Multimodal Machine Learning ApplicationsAdvanced Neural Network ApplicationsVisual Attention and Saliency Detection

Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learning

Open access
You, Zuyao, Wu, Zuxuan·2025·arXiv
DOI 10.48550/arxiv.2506.22624arXiv 2506.22624
Open this paper in CoChat — free accountOpen PDF

Abstract

We present Seg-R1, a preliminary exploration of using reinforcement learning (RL) to enhance the pixel-level understanding and reasoning capabilities of large multimodal models (LMMs). Starting with foreground segmentation tasks, specifically camouflaged object detection (COD) and salient object detection (SOD), our approach enables the LMM to generate point and bounding box prompts in the next-token fashion, which are then used to guide SAM2 in producing segmentation masks. We introduce Group Relative Policy Optimization (GRPO) into the segmentation domain, equipping the LMM with pixel-level comprehension through a carefully designed training strategy. Notably, Seg-R1 achieves remarkable performance with purely RL-based training, achieving .873 S-measure on COD10K without complex model modification. Moreover, we found that pure RL training demonstrates strong open-world generalization. Despite being trained solely on foreground segmentation image-mask pairs without text supervision, Seg-R1 achieves impressive zero-shot performance on referring segmentation and reasoning segmentation tasks, with 71.4 cIoU on RefCOCOg test and 56.7 gIoU on ReasonSeg test, outperforming models fully supervised on these datasets.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2506.22624

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing RLVR frameworks for segmentation focus predominantly on terminal outcomes and neglect the efficiency and logical coherence of intermediate actions.

    “Existing RLVR frameworks (You & Wu, 2025;Huang et al., 2025a;Liu et al., 2025b;Jiang et al., 2025b;Liu et al., 2025c;Su et al., 2025a) focus predominantly on terminal outcomes, such as the final segmentation accuracy (e.g., Dice or IoU), while neglecting the efficiency and logical coherence of intermediate actions.”

    From MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning (2026) · cites this paper as You & Wu, 2025

    Section 3.1 (p. 4) describes Seg-R1’s rewards as output-format compliance and segmentation quality measured by IoU and S-Measure, supporting its use as an example of outcome-focused segmentation RL. The paper does not establish that existing RLVR frameworks generally neglect the efficiency or logical coherence of intermediate actions. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Seg-R1 significantly outperforms other general-purpose MLLMs on medical datasets.

    “In contrast, although Seg-R1 (You & Wu, 2025) was not specifically fine-tuned on medical domain data, it significantly outperforms other general-purpose MLLMs.”

    From MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning (2026) · cites this paper as You & Wu, 2025

    The Introduction (p. 2) says Seg-R1 was trained on foreground-segmentation image-mask pairs, and the comparison results (p. 8) show it outperforming several models on ReasonSeg. But our copy does not report a comparison on medical datasets, so it does not support extending that result to the medical setting. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Existing MLLM-based interactive segmentation methods rely on single-turn, rigid interaction strategies.

    “Current studies either rely on single-turn prompts (You & Wu, 2025;Liu et al., 2025b;Huang et al., 2025a;Li et al., 2024), treating SAM as a static segmentor rather than an iterative agent, or adhere to rigid, point-only trajectories (Zhu et al., 2025;Jiang et al., 2026).”

    From MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning (2026) · cites this paper as You & Wu, 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 67

  1. [1][1] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 , 2023. - · arXiv
  2. [2][2] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 , 2025. - · arXiv
  3. [3][3] K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao. Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195 , 2023. - · arXiv
  4. [5][5] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. CVPR , 2022. -
  5. [6][6] B. Cheng, A. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. NeurIPS , 2021. -
  6. [8][8] T. Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 , 2023. - · arXiv
  7. [9][9] D.-P. Fan, M.-M. Cheng, Y. Liu, T. Li, and A. Borji. Structure-measure: A new way to evaluate foreground maps. In ICCV , 2017. -
  8. [10][10] D.-P. Fan, C. Gong, Y. Cao, B. Ren, M.-M. Cheng, and A. Borji. Enhanced-alignment measure for binary foreground map evaluation. arXiv preprint arXiv:1805.10421 , 2018. - · arXiv
  9. [11][11] D.-P. Fan, G.-P. Ji, G. Sun, M.-M. Cheng, J. Shen, and L. Shao. Camouflaged object detection. In CVPR , 2020. -
  10. [12][12] C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 , 2023. - · arXiv
  11. [14][14] J. Hu, Z. Cheng, and S. Gong. Int: Instance-specific negative mining for task-generic promptable segmentation. arXiv preprint arXiv:2501.18753 , 2025. - · arXiv
  12. [15][15] J. Hu, J. Lin, S. Gong, and W. Cai. Relax image-specific prompt requirement in sam: A single generic prompt for segmenting camouflaged objects. In AAAI , 2024. -
  13. [16][16] J. Hu, J. Lin, J. Yan, and S. Gong. Leveraging hallucinations to reduce manual prompt dependency in promptable segmentation. arXiv preprint arXiv:2408.15205 , 2024. - · arXiv
  14. [17][17] Z. Huang, H. Dai, T.-Z. Xiang, S. Wang, H.-X. Chen, J. Qin, and H. Xiong. Feature shrinkage pyramid for camouflaged object detection with transformers. In CVPR , 2023. -
  15. [18][18] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP , 2014. -
  16. [19][19] A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi. A diagram is worth a dozen images. In ECCV , 2016. -
  17. [20][20] A. Kirillov, K. He, R. Girshick, C. Rother, and P. Dollár. Panoptic segmentation. In CVPR , 2019. -
  18. [21][21] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. In ICCV , 2023. -
  19. [22][22] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP , 2023. -
  20. [23][23] X. Lai, Z. Tian, Y. Chen, Y. Li, Y. Yuan, S. Liu, and J. Jia. Lisa: Reasoning segmentation via large language model. In CVPR , 2024. -
  21. [25][25] G. Li and Y. Yu. Visual saliency based on multiscale deep features. In CVPR , 2015. -
  22. [26][26] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML , 2023. -
  23. [27][27] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML , pages 12888-12900. PMLR, 2022. -
  24. [28][28] Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 , 2023. - · arXiv
  25. [29][29] F. Liang, B. Wu, X. Dai, K. Li, Y. Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR , 2023. -
  26. [30][30] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014. -
  27. [31][31] C. Liu, H. Ding, and X. Jiang. Gres: Generalized referring expression segmentation. In CVPR , 2023. -
  28. [32][32] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. NeurIPS , 2023. -
  29. [33][33] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV , 2024. -
  30. [34][34] W. Liu, X. Shen, C.-M. Pun, and X. Cun. Explicit visual prompting for low-level structure segmentations. In CVPR , 2023. -
  31. [35][35] W. Liu, X. Shen, C.-M. Pun, and X. Cun. Explicit visual prompting for universal foreground segmentations. arXiv preprint arXiv:2305.18476 , 2023. - · arXiv
  32. [36][36] Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV , 2024. -
  33. [38][38] M. Ma, C. Xia, C. Xie, X. Chen, and J. Li. Receptive field broadening and boosting for salient object detection. arXiv preprint arXiv:2110.07859 , 2021. - · arXiv
  34. [39][39] H. Mei, G.-P. Ji, Z. Wei, X. Yang, X. Wei, and D.-P. Fan. Camouflaged object segmentation with distraction mining. In CVPR , 2021. -
  35. [42][42] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. NeurIPS , 2022. -
  36. [43][43] Y. Pang, X. Zhao, T.-Z. Xiang, L. Zhang, and H. Lu. Zoom in and out: A mixed-scale triplet network for camouflaged object detection. In CVPR , 2022. -
  37. [44][44] Y. Pang, X. Zhao, T.-Z. Xiang, L. Zhang, and H. Lu. Zoomnext: A unified collaborative pyramid network for camouflaged object detection. TPAMI , 2024. -
  38. [46][46] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV , 2015. -
  39. [47][47] X. Qin, H. Dai, X. Hu, D.-P. Fan, L. Shao, and L. V. Gool. Highly accurate dichotomous image segmentation. In ECCV , 2022. -
  40. [48][48] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. NeurIPS , 2023. -
  41. [49][49] H. Rasheed, M. Maaz, S. Shaji, A. Shaker, S. Khan, H. Cholakkal, R. M. Anwer, E. Xing, M.-H. Yang, and F. S. Khan. Glamm: Pixel grounding large multimodal model. In CVPR , pages 13009-13018, 2024. -
  42. [50][50] N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C.-Y. Wu, R. Girshick, P. Dollár, and C. Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 , 2024. - · arXiv
  43. [52][52] Z. Ren, Z. Huang, Y. Wei, Y. Zhao, D. Fu, J. Feng, and X. Jin. Pixellm: Pixel reasoning with large multimodal model. In CVPR , pages 26374-26383, 2024. -
  44. [53][53] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. - · arXiv
  45. [54][54] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 , 2024. - · arXiv
  46. [55][55] H. Shen, P. Liu, J. Li, C. Fang, Y. Ma, J. Liao, Q. Shen, Z. Zhang, K. Zhao, Q. Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615 , 2025. - · arXiv
  47. [56][56] J. Shi, Q. Yan, L. Xu, and J. Jia. Hierarchical image saliency detection on extended cssd. TPAMI , 2015. -
  48. [57][57] R. S. Sutton, A. G. Barto, et al. Reinforcement learning: An introduction . MIT press Cambridge, 1998. -
  49. [58][58] J. Wang, Z. Tian, X. Wang, X. Zhang, W. Huang, Z. Wu, and Y.-G. Jiang. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv preprint arXiv:2504.11455 , 2025. - · arXiv
  50. [59][59] L. Wang, H. Lu, Y. Wang, M. Feng, D. Wang, B. Yin, and X. Ruan. Learning to detect salient objects with image-level supervision. In CVPR , 2017. -
  51. [60][60] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191 , 2024. - · arXiv
  52. [61][61] C. Yang, L. Zhang, H. Lu, X. Ruan, and M.-H. Yang. Saliency detection via graph-based manifold ranking. In CVPR , 2013. -
  53. [62][62] Q. Ye, H. Xu, G. Xu, J. Ye, M. Yan, Y. Zhou, J. Wang, A. Hu, P. Shi, Y. Shi, et al. mplugowl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178 , 2023. - · arXiv
  54. [63][63] Z. You, L. Kong, L. Meng, and Z. Wu. FOCUS: Towards universal foreground segmentation. In AAAI , 2025. -
  55. [64][64] Z. You, J. Wang, L. Kong, B. He, and Z. Wu. Pix2cap-coco: Advancing visual comprehension via pixel-level captioning. arXiv preprint arXiv:2501.13893 , 2025. - · arXiv
  56. [65][65] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context in referring expressions. In ECCV , 2016. -
  57. [66][66] H. Yuan, X. Li, T. Zhang, Z. Huang, S. Xu, S. Ji, Y. Tong, L. Qi, J. Feng, and M.-H. Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.04001 , 2025. - · arXiv
  58. [67][67] Y. Yuan, W. Liu, P. Gao, Q. Dai, and J. Qin. Unified unsupervised salient object detection via knowledge transfer. arXiv preprint arXiv:2404.14759 , 2024. - · arXiv
  59. [69][69] S. Zhang, P. Sun, S. Chen, M. Xiao, W. Shao, W. Zhang, Y. Liu, K. Chen, and P. Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. In ECCV , 2025. -
  60. [70][70] T. Zhang, X. Li, H. Fei, H. Yuan, S. Wu, S. Ji, C. C. Loy, and S. Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. NeurIPS , 2024. -
  61. [71][71] T. Zhang, X. Li, Z. Huang, Y. Li, W. Lei, X. Deng, S. Chen, S. Ji, and J. Feng. Pixel-sail: Single transformer for pixel-grounded understanding. arXiv preprint arXiv:2504.10465 , 2025. - · arXiv
  62. [72][72] P. Zheng, D. Gao, D.-P. Fan, L. Liu, J. Laaksonen, W. Ouyang, and N. Sebe. Bilateral reference for high-resolution dichotomous image segmentation. arXiv preprint arXiv:2401.03407 , 2024. - · arXiv
  63. [73][73] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. In CVPR , 2017. -
  64. [75][75] H. Zhou, B. Qiao, L. Yang, J. Lai, and X. Xie. Texture-guided saliency distilling for unsupervised salient object detection. In CVPR , 2023. -
  65. [76][76] H. Zhu, P. Li, H. Xie, X. Yan, D. Liang, D. Chen, M. Wei, and J. Qin. I can find you! boundaryguided separated attention network for camouflaged object detection. In AAAI , 2022. -
  66. [77][77] X. Zou, Z.-Y. Dou, J. Yang, Z. Gan, L. Li, C. Li, X. Dai, H. Behl, J. Wang, L. Yuan, et al. Generalized decoding for pixel, image, and language. In CVPR , 2023. -
  67. [78][78] X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Wang, L. Wang, J. Gao, and Y. J. Lee. Segment everything everywhere all at once. NeurIPS , 2023.

Cited by

Sorted by citations

Related

Authors
You, Z., & Wu, Z.
ORCID
You, Zuyao
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2506.22624
arXiv
2506.22624
OpenAlex
W6948315201
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-sa/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
Keywords
large multimodal models, foreground segmentation, camouflaged object detection, salient object detection, Group Relative Policy Optimization, SAM2, zero-shot segmentation, referring segmentation, open-world generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review